---
title: "Ceasefire at FIGNEWS 2024 Shared Task: Automated Detection"
description: "The authors took part in FIGNEWS 2024 Subtask 1, which asks teams to sort news… 99% accuracy. 10 references, cited by 1 paper here."
authors: ["Noor Sadiah", "Sara Al-Emadi", "Sumaya Rahman"]
year: 2024
venue: "Proceedings of the Second Arabic Natural Language Processing Conference"
source: "https://aclanthology.org/2024.arabicnlp-1.63/"
doi: "10.18653/v1/2024.arabicnlp-1.63"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sadiah-2024-ceasefire-fignews-2024-shared-task/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sadiah-2024-ceasefire-fignews-2024-shared-task/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/13/) / Ceasefire at FIGNEWS 2024 Shared Task: Automated Detection and Annotation of Media Bias Using Large Language Models

# Ceasefire at FIGNEWS 2024 Shared Task: Automated Detection and Annotation of Media Bias Using Large Language Models

**Authors:** Noor Sadiah, [Sara Al-Emadi](https://md.research.elamur.ai/free/en/authors/sara-al-emadi/) and Sumaya Rahman · **Year:** 2024 · **Venue:** [Proceedings of the Second Arabic Natural Language Processing Conference](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2024](https://md.research.elamur.ai/free/en/venues/arabicnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.arabicnlp-1.63/) · [PDF](https://aclanthology.org/2024.arabicnlp-1.63.pdf) · [DOI 10.18653/v1/2024.arabicnlp-1.63](https://doi.org/10.18653/v1/2024.arabicnlp-1.63) · [OpenAlex](https://openalex.org/W4402684066)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/sadiah-2024-ceasefire-fignews-2024-shared-task/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The authors took part in FIGNEWS 2024 Subtask 1, which asks teams to sort news headlines, articles and social media posts about the Israel war on Gaza (2023–2024) into seven bias categories. The three-person team at Hamad Bin Khalifa University first wrote shared annotation guidelines. They tested those guidelines on 100 hand-labelled statements and measured inter-annotator agreement. They then automated the labelling with GPT-3.5 Turbo, using a prompt that held category definitions, per-label examples and a short glossary of conflict-specific names such as Hamas, IDF and Netanyahu. The prompt was improved over several rounds by spot-checking the model's errors. Their internal random checks suggested about 99% accuracy. The official shared-task scores were much lower: 46.3 accuracy and 29.3 macro F1. The team also analysed about 1,800 labelled statements and found that labels biased against Palestine clearly outnumbered both other main labels. The paper is useful mainly as a case study of LLM-driven annotation of a subjective, culturally loaded label set. It shows that informal self-checks can greatly overstate quality.

## Key points
- The model was GPT-3.5 Turbo (OpenAI API) run at temperature 0.4. In-context examples came from 100 manually labelled statements, 40% of which were set aside for validation.
- Agreement among the three annotators was low: Fleiss Kappa was 0.256 over 200 statements labelled for IAA.
- On the official evaluation the system reached 46.3 accuracy, 29.3 macro F1 and 61.2 F1 on the bias dimension. The roughly 99% figure comes only from random internal checks.
- Across the first two batches (about 1,800 statements), roughly 40% were labelled biased against Palestine, 19% biased against Israel and 29% unbiased.
- Practical limits were speed and cost. Labelling 500 statements took about an hour on a closed-source paid model, which pushed the team to cut the in-prompt examples to 10.

## Interesting ideas
- Put a small domain glossary (who Hamas, IDF and Netanyahu are) directly into the classification prompt, because the model kept mislabelling statements whose meaning depended on knowing these actors.
- The gap between a 99% self-checked score and a 46% official score is a warning that random spot checks by the people who wrote the prompt are not real evaluation.
- The authors noticed that the LLM had a prior leaning of its own and told it to judge only from the given text. This separates the evidence in the text from the model's background opinions.
- Low human agreement (kappa 0.256) on a subjective label set sets a natural ceiling on how much any automatic labeller can be trusted.
- Turning observed model mistakes straight back into prompt clarifications, such as extra examples for confusable labels or spelling traps like Hamas versus Hummus, is a cheap error-driven repair loop.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): LLM teacher tiers could be given a compact, compiled entity sheet from our world knowledge with each request, so silver labels for frames and roles are not skewed by the teacher's guesses about named entities.
- **I2** (learn): every teacher-produced silver set should be scored against a held-out gold slice and a negative control that the prompt authors never saw before any accuracy claim is made.
- **I3** (frames): when an LLM is used to propose speech acts, implicature or irony, the instructions should require each judgement to cite tokens or tree nodes in the input, so that an unsupported answer can be detected and rejected.
- **I4** (answer): for evaluative questions such as "is this character fair?", human agreement should be measured first and the answer reported with its uncertainty rather than as a single confident verdict.
- **I5** (learn): the same loop could feed self-induced repair rules, with each recurring teacher error class becoming an explicit rule or example that is checked for regressions.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Hamad bin Khalifa University](https://md.research.elamur.ai/free/en/universities/hamad-bin-khalifa-university/) — University, Doha, Qatar · 141 works · 14,175 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 577 works

## Cite

Cite as: Noor Sadiah, Sara Al-Emadi and Sumaya Rahman. 2024. Ceasefire at FIGNEWS 2024 Shared Task: Automated Detection and Annotation of Media Bias Using Large Language Models. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sadiah2024ceasefire,
  title = {{Ceasefire at FIGNEWS 2024 Shared Task: Automated Detection and Annotation of Media Bias Using Large Language Models}},
  author = {Noor Sadiah, Sara Al-Emadi and Sumaya Rahman},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.63},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.63}
}
```

- [← previous BSC-LANGTECH at FIGNEWS 2024 Shared Task: Exploring Semi-Automatic Bias Annotation using Frame Analysis](https://md.research.elamur.ai/free/en/papers/fernandez-2024-bsc-langtech-fignews-2024-shared/)
- [next → Classifying Graphemes in English Words Through the Application of a Fuzzy Inference System](https://md.research.elamur.ai/free/en/papers/rose-2024-classifying-graphemes-english-words-through/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/13/)

## References

10 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Zhu-G. Zheng J. Zhang L. Huang X. Zhong T. Li S. Du H. Ker C. L. Hou, C. 2024. Prompt-based and fine-tuned gpt models for context-dependent and -independent deductive coding in social annotation. In Proceedings of the 14th International Conference on Learning Analytics and Knowledge (LAK ’24). Jatowt-A. [find](https://scholar.google.com/scholar?q=Zhu%2DG%2E+Zheng+J%2E+Zhang+L%2E+Huang+X%2E+Zhong+T%2E+Li+S%2E+Du+H%2E+Ker+C%2E+L%2E+Hou%2C+C%2E+2024%2E+Prompt%2Dbased+and+fine%2Dtuned+gpt+models+for+context%2Ddependent+and+%2Dindependent+deductive+coding+in+social+annotation%2E+In+P)
2. Färber M. Yoshikawa M. Lim, S. 2020. Annotating and analyzing biased sentences in news articles using crowdsourcing. In Proceedings of the 12th International Conference on Language Resources and Evaluation (LREC 2020), pages 1487–1496. European Language Resources Association (ELRA). Yung-F. [find](https://scholar.google.com/scholar?q=F%C3%A4rber+M%2E+Yoshikawa+M%2E+Lim%2C+S%2E+2020%2E+Annotating+and+analyzing+biased+sentences+in+news+articles+using+crowdsourcing%2E+In+Proceedings+of+the+12th+International+Conference+on+Language+Resources+and+Evalu)
3. Scholman M. Tsarfaty R. Dagan I. Demberg V. Pyatkin, V. 2023. [Design choices for crowdsourcing implicit discourse relations: Revealing the biases introduced by task design](https://md.research.elamur.ai/free/en/papers/pyatkin-2023-design-choices-crowdsourcing-implicit-discourse/). arXiv. Rahman-M. [↗](https://doi.org/10.1162/tacl_a_00586)
4. Ghughe S. Raza, S. 2024. Dataset annotation and model building for identifying biases in news narratives. To be published. Rudnitckaia-L. [find](https://scholar.google.com/scholar?q=Ghughe+S%2E+Raza%2C+S%2E+2024%2E+Dataset+annotation+and+model+building+for+identifying+biases+in+news+narratives%2E+To+be+published%2E+Rudnitckaia%2DL%2E)
5. Mitrović J. Hamborg F. Granitzer M. Gipp B. Donnay K. Spinde, T. 2021. Automated identification of bias inducing words in news articles using linguistic and context-oriented features. Information Processing & Management, 58(3):102505. [↗](https://doi.org/10.1016/j.ipm.2021.102505)
6. P. Törnberg. 2024. Best practices for text annotation with large language models. arXiv. Chen-X. [↗](https://doi.org/10.48550/arxiv.2402.05129)
7. Xu N. Zu C. Shao Z. Liu S. Cui Y. Zhou Z. Gong C. Shen Y. Zhou J. Chen S. Gui T. Zhang Q. Huang X. Ye, J. 2023. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv. [↗](https://doi.org/10.48550/arxiv.2303.10420)
8. Wajdi Zaghouani, Mustafa Jarrar, Nizar Habash, Houda Bouamor, Imed Zitouni, Mona Diab, Samhaa R. El-Beltagy, and Muhammed AbuOdeh, editors. 2024. [The FIGNEWS Shared Task on News Media Narratives](https://md.research.elamur.ai/free/en/papers/zaghouani-2024-fignews-shared-task-news-media/). Association for Computational Linguistics, Bangkok, Thailand. Mi-F.
9. Meng H. Deng J. Zhou, J. 2022. Overview of nlpcc 2022 shared task 7: Fine-grained dialogue social bias measurement. In Proceedings of the NLPCC 2022, Lecture Notes in Computer Science, pages 31–40. Springer. Yin-Z. [find](https://scholar.google.com/scholar?q=Meng+H%2E+Deng+J%2E+Zhou%2C+J%2E+2022%2E+Overview+of+nlpcc+2022+shared+task+7%3A+Fine%2Dgrained+dialogue+social+bias+measurement%2E+In+Proceedings+of+the+NLPCC+2022%2C+Lecture+Notes+in+Computer+Science%2C+pages+31%E2%80%9340%2E+Sp)
10. Tyson G. Haq E. Lee L. Hui P. Zhu, Y. 2024. Apt-pipe: A prompt-tuning tool for social data annotation using chatgpt. Computation and Language. [find](https://scholar.google.com/scholar?q=Tyson+G%2E+Haq+E%2E+Lee+L%2E+Hui+P%2E+Zhu%2C+Y%2E+2024%2E+Apt%2Dpipe%3A+A+prompt%2Dtuning+tool+for+social+data+annotation+using+chatgpt%2E+Computation+and+Language%2E)

*Updated 1 November 2026*