---
title: "Narrative Navigators at FIGNEWS 2024 Shared Task: New"
description: "The paper describes how the team Narrative Navigators took part in the FIGNEWS 2024… 12.8 accuracy. 20 references, cited by 1 paper here."
authors: ["Maryam AlEmadi", "Jana ElMesselmani", "Lyna Bermak", "Goumana Abdullah", "Esra'a Sharqawi", "Anissa Jrad", "Zied Zouabi", "Wajdi Zaghouani"]
year: 2024
venue: "Proceedings of the Second Arabic Natural Language Processing Conference"
source: "https://aclanthology.org/2024.arabicnlp-1.57/"
doi: "10.18653/v1/2024.arabicnlp-1.57"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/alemadi-2024-narrative-navigators-fignews-2024-shared/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/alemadi-2024-narrative-navigators-fignews-2024-shared/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/13/) / Narrative Navigators at FIGNEWS 2024 Shared Task: New Frontiers in Bias and Propaganda Annotation Techniques

# Narrative Navigators at FIGNEWS 2024 Shared Task: New Frontiers in Bias and Propaganda Annotation Techniques

**Authors:** Maryam AlEmadi, Jana ElMesselmani, Lyna Bermak, Goumana Abdullah, Esra'a Sharqawi, Anissa Jrad, Zied Zouabi and [Wajdi Zaghouani](https://md.research.elamur.ai/free/en/authors/wajdi-zaghouani/) · **Year:** 2024 · **Venue:** [Proceedings of the Second Arabic Natural Language Processing Conference](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2024](https://md.research.elamur.ai/free/en/venues/arabicnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.arabicnlp-1.57/) · [PDF](https://aclanthology.org/2024.arabicnlp-1.57.pdf) · [DOI 10.18653/v1/2024.arabicnlp-1.57](https://doi.org/10.18653/v1/2024.arabicnlp-1.57) · [OpenAlex](https://openalex.org/W4402684307)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/alemadi-2024-narrative-navigators-fignews-2024-shared/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper describes how the team Narrative Navigators took part in the FIGNEWS 2024 Shared Task, which asked participants to manually label bias and propaganda in multilingual news posts about the 2023–2024 Israel-Palestine war. The posts came from official Facebook pages of news outlets. Seven annotators labelled 2,200 posts for each of two subtasks. The first subtask was the direction of bias: against Palestine, against Israel, or unbiased. The second was the presence of propaganda.

The team wrote its own guidelines with worked examples and trained annotators in a group session and then one-on-one. A shared spreadsheet logged changes to the guidelines. Agreement was checked daily with Cohen's Kappa, and ChatGPT was used as a helper with cross-verification.

On two tracks, Bias Guidelines and Bias Consistency, the team ranked third. Agreement was modest for bias (Kappa 39.4) and very low for propaganda (12.8). These numbers show how subjective such labels are, and the authors argue that better tools and stronger media literacy are needed.

## Key points
- The Bias Guidelines score was 0.7497. It combined a full Document Score (8/8) with an IAA Kappa of 39.4. The Consistency track measured how central a team's labels were relative to the other teams.
- Bias subtask results: accuracy 56.5%, Macro F1 45.5%, binary F1 Bias* 70.5%.
- Propaganda subtask results: Kappa 12.8, accuracy 54.5%, Macro F1 54.6%, F1 Prop* 63.1%. The team did not place in the top three here.
- Data volume: 2,200 items per subtask, split into 2 main batches and 4 IAA sets. The top team, NLPColab, reached a bias Kappa of 78.8.
- Prevalence in the team's own data: 42% biased against Palestine, 15% against Israel, 68% propaganda. Across 129,800 data points from all teams: 29.2%, 10.9% and 46.1%.

## Interesting ideas
- The Consistency track rewards a team for how close its labels sit to the consensus of all other teams, which turns disagreement among many independent labellers into a quality signal.
- The guidelines were kept as a living log: when new phenomena appeared, such as "Emotive Propaganda" or "Biased Terminology", the rules were updated and the change was recorded with its date.
- Results were reported both as binary detection (biased or not) and as fine-grained direction, which revealed that detection was much easier than deciding the direction.
- The LLM helper turned out to be unreliable on recent events because its knowledge was out of date, so its suggestions needed cross-checking.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Silver-data agreement could score each witness parser by its centrality among the other witnesses, so that outlier witnesses get less weight when votes are combined.
- **I2** (learn): Self-induced repair rules could carry a similar changelog that records what triggered each rule, keeping the evolution of the rule set explainable and reversible.
- **I3** (frames): Irony and stance detection could be evaluated in two layers, first whether the device is present and then whom it targets, so that errors can be attributed to the right step.
- **I4** (learn): LLM teacher tiers could flag time-sensitive or world-knowledge-dependent items and route them to stronger verification instead of accepting the teacher's label directly.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Hamad bin Khalifa University](https://md.research.elamur.ai/free/en/universities/hamad-bin-khalifa-university/) — University, Doha, Qatar · 141 works · 14,175 citations
- [Carnegie Mellon University Qatar](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university-qatar/) — University, Doha, Qatar · 47 works · 2,773 citations
- [Northwestern University in Qatar](https://md.research.elamur.ai/free/en/universities/northwestern-university-in-qatar/) — University, Doha, Qatar · 24 works · 148 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 577 works

## Cite

Cite as: Maryam AlEmadi, Jana ElMesselmani, Lyna Bermak, Goumana Abdullah, Esra'a Sharqawi, Anissa Jrad, Zied Zouabi and Wajdi Zaghouani. 2024. Narrative Navigators at FIGNEWS 2024 Shared Task: New Frontiers in Bias and Propaganda Annotation Techniques. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{alemadi2024narrative,
  title = {{Narrative Navigators at FIGNEWS 2024 Shared Task: New Frontiers in Bias and Propaganda Annotation Techniques}},
  author = {Maryam AlEmadi, Jana ElMesselmani, Lyna Bermak, Goumana Abdullah, Esra'a Sharqawi, Anissa Jrad, Zied Zouabi and Wajdi Zaghouani},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.57},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.57}
}
```

- [← previous muNERa at WojoodNER 2024: Multi-tasking NER Approach](https://md.research.elamur.ai/free/en/papers/alotaibi-2024-munera-wojoodner-2024-multi-tasking/)
- [next → Neo-Assyrian Metaphors through the Telescope: Linguistic Patterns involving Body Part Constructions in the State Archives Letter Corpus](https://md.research.elamur.ai/free/en/papers/ong-2024-neo-assyrian-metaphors-through-telescope/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/13/)

## References

20 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Chiraz El-Khaissi, Kartikeya Siddhant, AbdelRahim Elmadany, Sarah Jeblee, Maryam Al-Khouja, Motaz Saad, Muhammad Abdul-Mageed, and Kareem Darwish. 2022. [Detecting Offensive Language on Arabic Twitter](https://md.research.elamur.ai/free/en/papers/alam-2022-overview-wanlp-2022-shared-task/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 105-118. Association for Computational Linguistics. https://aclanthology.org/2022.wanlp-1.11.pdf. [↗](https://aclanthology.org/2022.wanlp-1.11/)
2. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Giovanni Da San Martino, and Preslav Nakov. 2022. [Overview of the WANLP 2022 Shared Task on Propaganda Detection in Arabic](https://md.research.elamur.ai/free/en/papers/alam-2022-overview-wanlp-2022-shared-task/). arXiv preprint arXiv:2211.10057. Gavin Davie. Gatekeeping Theory. Mass Communication Theory: from Theory to Practical Application. [Online]. Available: https://masscommtheory.com/theory-overviews/gatekeeping-theory/ [Accessed: 4 th May 2024]. [↗](https://arxiv.org/abs/2211.10057)
3. Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019. [Fine-Grained Analysis of Propaganda in News Article](https://md.research.elamur.ai/free/en/papers/martino-2019-fine-grained-analysis-propaganda-news/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5636–5646, Hong Kong, China. Association for Computational Linguistics. DOI: 10.18653/v1/D19-1565. Available: https://aclanthology.org/D19-1565. [↗](https://doi.org/10.18653/v1/d19-1565)
4. Konstantina Lazaridou, Alexander Löser, Maria Mestre, and Felix Naumann. 2020. Discovering Biased News Articles: Leveraging Multiple Human Annotations. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1268–1277, Marseille, France, 11–16 May 2020. European Language Resources Association (ELRA). [CC-BY-NC]. Available: https://aclanthology.org/2020.lrec-1.159.pdf. [↗](https://aclanthology.org/2020.lrec-1.159/)
5. Luke Auburn. 2023. Study of headlines shows media bias is growing. University of Rochester News Center. [Online].Available: [https://www.rochester.edu/newscenter/study-of-headlines-shows-media-bias-growing-563502/](https://www.rochester.edu/newscenter/study-of-headlines-shows-media-bias-growing-563502/) [Accessed: 3 rd May 2024]. [↗](https://www.rochester.edu/newscenter/study-of-headlines-shows-media-bias-growing-563502/)
6. Marc Owen Jones. 2019. The Gulf Information War| Propaganda, Fake News, and Fake Trends: The Weaponization of Twitter Bots in the Gulf Crisis. International Journal of Communication, vol. 13. [Online]. Available: [http://ijoc.org/index.php/ijoc/article/viewFile/XXX](http://ijoc.org/index.php/ijoc/article/viewFile/XXX) XX/XXXXX. [Accessed: 4 th May 2024]. [↗](http://ijoc.org/index.php/ijoc/article/viewFile/XXX)
7. Marc Owen Jones. 2024. Twitter post. April 12. "Full text of the tweet [6/ It's disturbing that @TheAtlantic would allow these two crucially important aspects of Israeli propaganda (and indeed, misinformation), to be published. By all means criticize Hamas, but not on the basis of non-credible or plausible narratives. I also find it bizarre that]." [Online]. Available: [https://twitter.com/marcowenjones/status/1778753](https://twitter.com/marcowenjones/status/1778753) 389817409966. [Accessed: 4th May 2024]. [↗](https://twitter.com/marcowenjones/status/1778753)
8. Richard Paul and Linda Elder. 2004. How to Detect Media Bias & Propaganda In National and World News. (Original work published June 30, 2004). [Online]. Available: [https://www.criticalthinking.org/TGS_files/SAM-MediaBias.pdf](https://www.criticalthinking.org/TGS_files/SAM-MediaBias.pdf) [Accessed: 10 th May 2024]. [↗](https://www.criticalthinking.org/TGS_files/SAM-MediaBias.pdf)
9. Shaina Raza, Mizanur Rahman, and Shardul Ghughe. 2024. Dataset Annotation and Model Building for Identifying Biases in News Narratives. In Text2Story 2024: Seventh International Workshop on Narrative Extraction from Texts held in conjunction with the 46th European Conference on Information Retrieval, Glasgow, Scotland. [Online]. Available: [https://ceur-ws.org/Vol-3671/paper1.pdf](https://ceur-ws.org/Vol-3671/paper1.pdf). [Accessed: 10 th May 2024]. [↗](https://ceur-ws.org/Vol-3671/paper1.pdf)
10. Timo Spinde, Manuel Plank, Jan-David Krieger, Terry Ruas, Bela Gipp, and Akiko Aizawa. 2021. Neural Media Bias Detection Using Distant Supervision With BABE - Bias Annotations By Experts. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1166–1177, Punta Cana, Dominican Republic. Association for Computational Linguistics. DOI: 10.18653/v1/2021.findings-emnlp.101. Available: https://aclanthology.org/2021.findings-emnlp.101. [↗](https://doi.org/10.18653/v1/2021.findings-emnlp.101)
11. Wajdi Zaghouani and Anis Charfi. 2018. Annotation Guidelines for Text Analytics in Social Media. In Qatar Foundation Annual Research Conference Proceedings, Vol. 2018, Issue 3, Mar 2018, ICTPD879. Hamad bin Khalifa University Press (HBKU Press). DOI: https://doi.org/10.5339/qfarc.2018.ICTPD879. [↗](https://doi.org/10.5339/qfarc.2018.ictpd879)
12. Wajdi Zaghouani and Anis Charfi. 2018. [Guidelines and Annotation Framework for Arabic Author Profiling](https://md.research.elamur.ai/free/en/papers/zaghouani-2018-guidelines-annotation-framework-arabic-author/). In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools. http://lrec-conf.org/workshops/lrec2018/W30/pdf/5_W30.pdf [↗](https://doi.org/10.48550/arxiv.1808.07678)
13. Wajdi Zaghouani and Dina Awad. 2016. Toward an Arabic Punctuated Corpus: Annotation Guidelines and Evaluation. In Proceedings of the 2nd Workshop on Arabic Corpora and Processing Tools 2016 Theme: Social Media, page 22. [find](https://scholar.google.com/scholar?q=Wajdi+Zaghouani+and+Dina+Awad%2E+2016%2E+Toward+an+Arabic+Punctuated+Corpus%3A+Annotation+Guidelines+and+Evaluation%2E+In+Proceedings+of+the+2nd+Workshop+on+Arabic+Corpora+and+Processing+Tools+2016+Theme%3A+Soc)
14. Wajdi Zaghouani, Behrang Mohit, Nizar Habash, Ossama Obeid, Nadi Tomeh, Alla Rozovskaya, Noura Farra, Sarah Alkuhlani, and Kemal Oflazer. 2014. [Large Scale Arabic Error Annotation: Guidelines and Framework](https://md.research.elamur.ai/free/en/papers/zaghouani-2014-large-scale-arabic-error-annotation/). In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages not listed, Reykjavik, Iceland. European Language Resources Association (ELRA). Available: http://www.lrec-conf.org/proceedings/lrec2014/pdf/956_Paper.pdf. [↗](https://doi.org/10.13140/rg.2.1.3273.6169)
15. Wajdi Zaghouani, Houda Bouamor, Abdelati Hawwari, Mona Diab, Ossama Obeid, Mahmoud Ghoneim, Sawsan Alqahtani, and Kemal Oflazer. 2016. [Guidelines and Framework for a Large Scale Arabic Diacritized Corpus](https://md.research.elamur.ai/free/en/papers/zaghouani-2016-guidelines-framework-large-scale-arabic/). In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 3637–3643.
16. Wajdi Zaghouani, Mustafa Jarrar, Nizar Habash, Houda Bouamor, Imed Zitouni, Mona Diab, Samhaa R. El-Beltagy, and Muhammed AbuOdeh. 2024. [The FIGNEWS Shared Task on News Media Narratives](https://md.research.elamur.ai/free/en/papers/zaghouani-2024-fignews-shared-task-news-media/). In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Association for Computational Linguistics, Bangkok, Thailand. [↗](https://doi.org/10.48550/arxiv.2407.18147)
17. Wajdi Zaghouani, Nizar Habash, Behrang Mohit, Abeer Heider, Alla Rozovskaya, and Kemal Oflazer. 2014. Annotation Guidelines For Non-native Arabic Text In The Qatar Arabic Language Bank. In Qatar Foundation Annual Research Conference Proceedings, Vol. 2014, Issue 1, Nov 2014, ITPP1105. Hamad bin Khalifa University Press (HBKU Press). DOI: 10.5339/qfarc.2014.ITPP1105. [↗](https://doi.org/10.5339/qfarc.2014.itpp1105)
18. Wajdi Zaghouani, Nizar Habash, Houda Bouamor, Alla Rozovskaya, Behrang Mohit, Abeer Heider, and Kemal Oflazer. 2015. [Correction Annotation for Non-Native Arabic Texts: Guidelines and Corpus](https://md.research.elamur.ai/free/en/papers/zaghouani-2015-correction-annotation-non-native-arabic/). In Proceedings of the 9th Linguistic Annotation Workshop, pages 129–139, Denver, Colorado, USA. Association for Computational Linguistics. DOI: 10.3115/v1/W15-1614. Available: https://aclanthology.org/W15-1614. [↗](https://doi.org/10.3115/v1/w15-1614)
19. Yahya R. Kamalipour. 2010. Language, media and war: Manipulating public perceptions. Media, 3(1-4):87. [find](https://scholar.google.com/scholar?q=Yahya+R%2E+Kamalipour%2E+2010%2E+Language%2C+media+and+war%3A+Manipulating+public+perceptions%2E+Media%2C+3%281%2D4%29%3A87%2E)
20. Yang Xiao and Firoj Alam. 2023. [Nexus at ArAIEval Shared Task: Fine-tuning Arabic language models for propaganda and disinformation detection](https://md.research.elamur.ai/free/en/papers/xiao-2023-nexus-araieval-shared-task-fine/). arXiv preprint arXiv:2311.03184. Available: https://arxiv.org/abs/2311.03184. [↗](https://arxiv.org/abs/2311.03184)

*Updated 1 November 2026*