---
title: "GroningenAnnotatesGaza at the FIGNEWS 2024 Shared Task:"
description: "The paper describes how one team built an annotation scheme for media bias in coverage of the 2023 Gaza War, created for… 23 references."
authors: ["Khalid Khatib", "Sara Gemelli", "Saskia Heisterborg", "Pritha Majumdar", "Gosse Minnema", "Arianna Muti", "Noa Solissa"]
year: 2024
venue: "Proceedings of the Second Arabic Natural Language Processing Conference"
source: "https://aclanthology.org/2024.arabicnlp-1.68/"
doi: "10.18653/v1/2024.arabicnlp-1.68"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/khatib-2024-groningenannotatesgaza-fignews-2024-shared-task/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/khatib-2024-groningenannotatesgaza-fignews-2024-shared-task/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/16/) / GroningenAnnotatesGaza at the FIGNEWS 2024 Shared Task: Analyzing Bias in Conflict Narratives

# GroningenAnnotatesGaza at the FIGNEWS 2024 Shared Task: Analyzing Bias in Conflict Narratives

**Authors:** [Khalid Khatib](https://md.research.elamur.ai/free/en/authors/khalid-al-khatib/), Sara Gemelli, Saskia Heisterborg, Pritha Majumdar, Gosse Minnema, [Arianna Muti](https://md.research.elamur.ai/free/en/authors/arianna-muti/) and Noa Solissa · **Year:** 2024 · **Venue:** [Proceedings of the Second Arabic Natural Language Processing Conference](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2024](https://md.research.elamur.ai/free/en/venues/arabicnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.arabicnlp-1.68/) · [PDF](https://aclanthology.org/2024.arabicnlp-1.68.pdf) · [DOI 10.18653/v1/2024.arabicnlp-1.68](https://doi.org/10.18653/v1/2024.arabicnlp-1.68) · [OpenAlex](https://openalex.org/W4402671192)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper describes how one team built an annotation scheme for media bias in coverage of the 2023 Gaza War, created for sub-task 1 of the FIGNEWS 2024 shared task. The data are multilingual news headlines and Facebook posts from news outlets in English, Arabic, Hebrew, French and Hindi. Seven annotators followed the prescriptive paradigm. They ran two pilot rounds of 50 items each, discussed the cases they disagreed on, and wrote each resolution into the guidelines. The settled cases cover quotes, hashtags, off-topic posts and texts whose bias could point either way. The team also added a partial layer of narrative frames that leaned pro-Israel or pro-Palestine, used to support the bias decision. They labelled 2000 items and reached Krippendorff's α = 0.420, which shows how subjective the task is. About half of the posts were judged unbiased. Among the biased posts, bias against Palestine was much more common than bias against Israel, and the rates differed sharply by language. The work matters because it gives concrete, reusable rules for annotating stance in contested news.

## Key points
- The team labelled 2000 instances in batches B01 and B02 using a seven-category scheme: four categories for the target of bias (Palestine, Israel, both, or others), plus categories for unbiased, unclear, and not applicable.
- Seven annotators reached Krippendorff's α of 0.420, and every annotator pair had observed agreement above 0.5.
- The abstract reports this label distribution: 52% unbiased, 42% against Palestine, 5% against Israel, 3% against both, and 16% unclear or not applicable.
- Hebrew texts had the fewest unbiased items (about 25%, versus 50–70% in the other languages) and the most bias against Palestine. French texts showed the most bias against Israel, and Hindi texts had the most "unclear" labels.
- The frame inventory has 5 frame categories for the Palestinian narrative and 8 for the Israeli narrative. The authors suggest replacing the "against X" labels with "pro-X" labels.

## Interesting ideas
- Bias in a quote is credited to the named speaker, not the author, so attributed quotes count as unbiased and unattributed ones become "not applicable". This separates the writer's stance from reported stance.
- Guidelines are built through small pilot rounds: annotators label a sample, gather disagreements, settle them by consensus, write the ruling into the guidelines, and run the next pilot.
- Narrative frames serve as a helper layer that informs the main label without deciding it. For example, a frame that glorifies one army can reveal bias even when the other side is never named.
- Hashtags and similar add-ons count only when they consistently signal a viewpoint or add content; otherwise the main text decides.
- The authors observe that a group such as Hamas, or a government, should not stand in for a whole country, and this affects how a label's target is chosen.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (frames): reported-speech frames could attach evaluative content to the source speaker's node, and give no holder when the source is missing, instead of assigning it to the narrator.
- **I2** (learn): when witness parsers or LLM teacher tiers disagree on silver data, the disputed cases can be logged, resolved once into an explicit repair rule, and rechecked on a fresh sample.
- **I3** (world): a perspective layer over the text world could record which narrative a passage supports, so implicit stance can be explained even when the affected party is absent from the text.
- **I4** (tokens): the tokenizer could tag hashtags and reach-boosting tails as a separate peripheral class, so they stay out of clause parsing but remain available as weak cues.
- **I5** (world): the ontology could keep organisations, governments and populations as distinct entities with part-of links, so stance aimed at one is not automatically passed to the others.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Groningen](https://md.research.elamur.ai/free/en/universities/university-of-groningen/) — University, Groningen, Netherlands · 74 works · 5,142 citations
- [University of Bologna](https://md.research.elamur.ai/free/en/universities/university-of-bologna/) — University, Bologna, Italy · 19 works · 2,067 citations
- [University of Bergamo](https://md.research.elamur.ai/free/en/universities/university-of-bergamo/) — University, Bergamo, Italy · 3 works · 75 citations
- [University of Pavia](https://md.research.elamur.ai/free/en/universities/university-of-pavia/) — University, Pavia, Italy · 7 works · 69 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 626 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works

## Cite

Cite as: Khalid Khatib, Sara Gemelli, Saskia Heisterborg, Pritha Majumdar, Gosse Minnema, Arianna Muti and Noa Solissa. 2024. GroningenAnnotatesGaza at the FIGNEWS 2024 Shared Task: Analyzing Bias in Conflict Narratives. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{khatib2024groningenannotatesgaza,
  title = {{GroningenAnnotatesGaza at the FIGNEWS 2024 Shared Task: Analyzing Bias in Conflict Narratives}},
  author = {Khalid Khatib, Sara Gemelli, Saskia Heisterborg, Pritha Majumdar, Gosse Minnema, Arianna Muti and Noa Solissa},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.68},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.68}
}
```

- [← previous GLARE: Google Apps Arabic Reviews Dataset](https://md.research.elamur.ai/free/en/papers/alghamdi-2024-glare-google-apps-arabic-reviews/)
- [next → Hate Speech Detection and Classification in Amharic Text with Deep Learning](https://md.research.elamur.ai/free/en/papers/gashe-2024-hate-speech-detection-classification-amharic/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/16/)

## References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yamen Ajjour, Milad Alshomary, Henning Wachsmuth, and Benno Stein. 2019. Modeling frames in argumentation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2922–2932, Hong Kong, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yamen+Ajjour%2C+Milad+Alshomary%2C+Henning+Wachsmuth%2C+and+Benno+Stein%2E+2019%2E+Modeling+frames+in+argumentation%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and)
2. Desislava Aleksandrova, François Lareau, and Pierre André Ménard. 2019. Multilingual sentence-level bias detection in Wikipedia. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 42–51, Varna, Bulgaria. INCOMA Ltd. [find](https://scholar.google.com/scholar?q=Desislava+Aleksandrova%2C+Fran%C3%A7ois+Lareau%2C+and+Pierre+Andr%C3%A9+M%C3%A9nard%2E+2019%2E+Multilingual+sentence%2Dlevel+bias+detection+in+Wikipedia%2E+In+Proceedings+of+the+International+Conference+on+Recent+Advances+in+Na)
3. Aishwarya Anegundi, Konstantin Schulz, Christian Rauh, and Georg Rehm. 2022. Modelling cultural and socio-economic dimensions of political bias in German tweets. In Proceedings of the 18th Conference on Natural Language Processing (KONVENS 2022), pages 29–40, Potsdam, Germany. KONVENS 2022 Organizers. [find](https://scholar.google.com/scholar?q=Aishwarya+Anegundi%2C+Konstantin+Schulz%2C+Christian+Rauh%2C+and+Georg+Rehm%2E+2022%2E+Modelling+cultural+and+socio%2Deconomic+dimensions+of+political+bias+in+German+tweets%2E+In+Proceedings+of+the+18th+Conference+)
4. Francesco Antici, Federico Ruggeri, Andrea Galassi, Katerina Korre, Arianna Muti, Alessandra Bardi, Alice Fedotova, and Alberto Barrón-Cedeño. 2024. [A corpus for sentence-level subjectivity detection on English news articles](https://md.research.elamur.ai/free/en/papers/antici-2024-corpus-sentence-level-subjectivity-detection/). In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 273–285, Torino, Italia. ELRA and ICCL.
5. Robert A Hackett. 1984. Decline of a paradigm? bias and objectivity in news media studies. Critical Studies in Media Communication, 1(3):229–259. [find](https://scholar.google.com/scholar?q=Robert+A+Hackett%2E+1984%2E+Decline+of+a+paradigm%3F+bias+and+objectivity+in+news+media+studies%2E+Critical+Studies+in+Media+Communication%2C+1%283%29%3A229%E2%80%93259%2E)
6. Felix Hamborg. 2023. Revealing Media Bias in News Articles: NLP Techniques for Automated Frame Analysis. [find](https://scholar.google.com/scholar?q=Felix+Hamborg%2E+2023%2E+Revealing+Media+Bias+in+News+Articles%3A+NLP+Techniques+for+Automated+Frame+Analysis%2E)
7. Christoph Hube and Besnik Fetahu. 2019. Neural based statement classification for biased language. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, WSDM ’19, page 195–203, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Christoph+Hube+and+Besnik+Fetahu%2E+2019%2E+Neural+based+statement+classification+for+biased+language%2E+In+Proceedings+of+the+Twelfth+ACM+International+Conference+on+Web+Search+and+Data+Mining%2C+WSDM+%E2%80%9919%2C+p)
8. Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019. [SemEval-2019 task 4: Hyperpartisan news detection](https://md.research.elamur.ai/free/en/papers/kiesel-2019-semeval-2019-task-4-hyperpartisan/). In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 829–839, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
9. Sora Lim, Adam Jatowt, Michael Färber, and Masatoshi Yoshikawa. 2020. Annotating and analyzing biased sentences in news articles using crowdsourcing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1478–1484. [find](https://scholar.google.com/scholar?q=Sora+Lim%2C+Adam+Jatowt%2C+Michael+F%C3%A4rber%2C+and+Masatoshi+Yoshikawa%2E+2020%2E+Annotating+and+analyzing+biased+sentences+in+news+articles+using+crowdsourcing%2E+In+Proceedings+of+the+Twelfth+Language+Resources+a)
10. Sora Lim, Adam Jatowt, and Masatoshi Yoshikawa. 2018. Understanding characteristics of biased sentences in news articles. In CIKM workshops, pages 121–128. [find](https://scholar.google.com/scholar?q=Sora+Lim%2C+Adam+Jatowt%2C+and+Masatoshi+Yoshikawa%2E+2018%2E+Understanding+characteristics+of+biased+sentences+in+news+articles%2E+In+CIKM+workshops%2C+pages+121%E2%80%93128%2E)
11. Siyi Liu, Lei Guo, Kate Mays, Margrit Betke, and Derry Tanti Wijaya. 2019. Detecting frames in news headlines and its application to analyzing news framing trends surrounding us gun violence. In Proceedings of the 23rd conference on computational natural language learning (CoNLL), pages 504–514. [find](https://scholar.google.com/scholar?q=Siyi+Liu%2C+Lei+Guo%2C+Kate+Mays%2C+Margrit+Betke%2C+and+Derry+Tanti+Wijaya%2E+2019%2E+Detecting+frames+in+news+headlines+and+its+application+to+analyzing+news+framing+trends+surrounding+us+gun+violence%2E+In+Proce)
12. Fred Morstatter, Liang Wu, Uraz Yavanoglu, Stephen R Corman, and Huan Liu. 2018. Identifying framing bias in online news. ACM Transactions on Social Computing, 1(2):1–18. [find](https://scholar.google.com/scholar?q=Fred+Morstatter%2C+Liang+Wu%2C+Uraz+Yavanoglu%2C+Stephen+R+Corman%2C+and+Huan+Liu%2E+2018%2E+Identifying+framing+bias+in+online+news%2E+ACM+Transactions+on+Social+Computing%2C+1%282%29%3A1%E2%80%9318%2E)
13. Barbara Plank. 2022. [The “problem” of human label variation: On ground truth in data, modeling and evaluation](https://md.research.elamur.ai/free/en/papers/plank-2022-problem-human-label-variation-ground/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2022.emnlp-main.731)
14. Robert M Entman. 1993. Framing: Toward clarification of a fractured paradigm. Journal of communication, 43(4):51–58. [find](https://scholar.google.com/scholar?q=Robert+M+Entman%2E+1993%2E+Framing%3A+Toward+clarification+of+a+fractured+paradigm%2E+Journal+of+communication%2C+43%284%29%3A51%E2%80%9358%2E)
15. Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. 2018. [A stylometric inquiry into hyperpartisan and fake news](https://md.research.elamur.ai/free/en/papers/potthast-2018-stylometric-inquiry-into-hyperpartisan-fake/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 231–240, Melbourne, Australia. Association for Computational Linguistics.
16. Yudi Guzmán-Monteza. 2023. Assessment of an annotation method for the detection of spanish argumentative, non-argumentative, and their components. Telematics and Informatics Reports, 11:100068. [find](https://scholar.google.com/scholar?q=Yudi+Guzm%C3%A1n%2DMonteza%2E+2023%2E+Assessment+of+an+annotation+method+for+the+detection+of+spanish+argumentative%2C+non%2Dargumentative%2C+and+their+components%2E+Telematics+and+Informatics+Reports%2C+11%3A100068%2E)
17. Daniel Preoţiuc-Pietro, Ye Liu, Daniel Hopkins, and Lyle Ungar. 2017. Beyond binary labels: Political ideology prediction of Twitter users. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 729–740, Vancouver, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Daniel+Preot%CC%A7iuc%2DPietro%2C+Ye+Liu%2C+Daniel+Hopkins%2C+and+Lyle+Ungar%2E+2017%2E+Beyond+binary+labels%3A+Political+ideology+prediction+of+Twitter+users%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Associatio)
18. Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. [Linguistic models for analyzing and detecting biased language](https://md.research.elamur.ai/free/en/papers/recasens-2013-linguistic-models-analyzing-detecting-biased/). In Proceedings of the 51st annual meeting of the Association for Computational Linguistics (volume 1: long papers), pages 1650–1659.
19. Paul Rottger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022. [Two contrasting data annotation paradigms for subjective NLP tasks](https://md.research.elamur.ai/free/en/papers/rottger-2022-two-contrasting-data-annotation-paradigms/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 175–190, Seattle, United States. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2022.naacl-main.13)
20. Matt Thomas, Bo Pang, and Lillian Lee. 2006. [Get out the vote: Determining support or opposition from congressional floor-debate transcripts](https://md.research.elamur.ai/free/en/papers/thomas-2006-get-out-vote-determining-support/). arXiv preprint cs/0607062.
21. Almar van der Stappen and Mathias Funk. 2021. Towards guidelines for designing human-in-the-loop machine training interfaces. In 26th International Conference on Intelligent User Interfaces, IUI ’21, page 514–519, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Almar+van+der+Stappen+and+Mathias+Funk%2E+2021%2E+Towards+guidelines+for+designing+human%2Din%2Dthe%2Dloop+machine+training+interfaces%2E+In+26th+International+Conference+on+Intelligent+User+Interfaces%2C+IUI+%E2%80%9921%2C+)
22. Francielle Vargas, Kokil Jaidka, Thiago Pardo, and Fabricio Benevenuto. 2023. Predicting sentence-level factuality of news and bias of media outlets. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2023), pages 1197–1206, Varna, Bulgaria. [find](https://scholar.google.com/scholar?q=Francielle+Vargas%2C+Kokil+Jaidka%2C+Thiago+Pardo%2C+and+Fabricio+Benevenuto%2E+2023%2E+Predicting+sentence%2Dlevel+factuality+of+news+and+bias+of+media+outlets%2E+In+Proceedings+of+the+International+Conference+on+)
23. Qi Yu. 2023. Towards a more in-depth detection of political framing. In Proceedings of the 7th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 162–174, Dubrovnik, Croatia. Association for Computational Linguistics. *Figure 2: The observed agreement per pair ranges from 0.54-0.77, with the highest score between annotators 5 and 6.* Wajdi Zaghouani, Mustafa Jarrar, Nizar Habash, Houda Bouamor, Imed Zitouni, Mona Diab, Samhaa R. El-Beltagy, and Muhammed AbuOdeh, editors. 2024. [The FIGNEWS Shared Task on News Media Narratives](https://md.research.elamur.ai/free/en/papers/zaghouani-2024-fignews-shared-task-news-media/). Association for Computational Linguistics, Bangkok, Thailand.

*Updated 2 November 2026*