---
title: "Bias Bluff Busters at FIGNEWS 2024 Shared Task: Developing"
description: "This paper describes one team's entry to FIGNEWS-2024, a shared task that asked participants to write… 9 references, cited by 1 paper here."
authors: ["Silvia Pareti", "Jasmin Heierli", "Serena Pareti", "Tatiana Lando"]
year: 2024
venue: "Proceedings of the Second Arabic Natural Language Processing Conference"
source: "https://aclanthology.org/2024.arabicnlp-1.62/"
doi: "10.18653/v1/2024.arabicnlp-1.62"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pareti-2024-bias-bluff-busters-fignews-2024/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pareti-2024-bias-bluff-busters-fignews-2024/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/13/) / Bias Bluff Busters at FIGNEWS 2024 Shared Task: Developing Guidelines to Make Bias Conscious

# Bias Bluff Busters at FIGNEWS 2024 Shared Task: Developing Guidelines to Make Bias Conscious

**Authors:** [Silvia Pareti](https://md.research.elamur.ai/free/en/authors/silvia-pareti/), Jasmin Heierli, Serena Pareti and [Tatiana Lando](https://md.research.elamur.ai/free/en/authors/tatiana-lando/) · **Year:** 2024 · **Venue:** [Proceedings of the Second Arabic Natural Language Processing Conference](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2024](https://md.research.elamur.ai/free/en/venues/arabicnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.arabicnlp-1.62/) · [PDF](https://aclanthology.org/2024.arabicnlp-1.62.pdf) · [DOI 10.18653/v1/2024.arabicnlp-1.62](https://doi.org/10.18653/v1/2024.arabicnlp-1.62) · [OpenAlex](https://openalex.org/W4402671171)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/pareti-2024-bias-bluff-busters-fignews-2024/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This paper describes one team's entry to FIGNEWS-2024, a shared task that asked participants to write annotation guidelines and then label news and social-media texts about the Gaza conflict for negative bias and propaganda. Israel and Palestine were the main targets, and every other target was grouped as "others." In about a month, four linguists went from a draft (v0.1) to finished guidelines (v1.0). The guidelines include:
- definitions of which entities count as Israel or Palestine,
- a glossary of connotated terms,
- linguistic cues for bias and propaganda,
- conventions for quotations and clickbait,
- a short set of diagnostic test questions.

Before starting, annotators held a session to name their own biases, and they wrote a justification for each label before assigning it. ChatGPT served as an extra annotator on the main set so that disagreements with humans could be found and arbitrated. Agreement among the experts was fair to moderate (α 0.43 for bias, 0.30 for propaganda). Once systematic errors were removed, ChatGPT agreed with the humans at about the same level as the humans agreed with each other. The authors argue that bias labels are inherently subjective. They also caution that high agreement only means something if the annotator pool is diverse.

## Key points
- The team single-annotated 1,800 Main texts and four-way annotated 200 IAA texts. Text length ran from 5 to over 1,400 words.
- Human IAA was Krippendorff's α 0.43 (Kappa 43.3) for bias and α 0.30 (Kappa 31.5) for propaganda.
- Raw human–ChatGPT agreement on bias was α 0.143. It rose to 0.315 after the team removed three kinds of excusable errors:
  - 64 cases with an invented label,
  - 1 case where the model misunderstood the request,
  - 788 cases where Hamas was treated as Palestine, a distinction the prompt never stated.
- For propaganda, human–ChatGPT agreement was α 0.346.
- In the Main set, labels were heavily skewed:
  - about one third Unbiased,
  - about one third bias against others, mostly Hamas,
  - about 10% against Israel,
  - about 2.5% against Palestine,
  - over 21% Unclear.
- The team proposed "Humanization" as a new propaganda technique: inserting personal victim details to build sympathy for one side.

## Interesting ideas
- Pairing a human annotator with an LLM and reviewing only the cases where they disagree concentrates expert effort on the hard items.
- The authors traced the largest block of LLM errors to a definition missing from the prompt (Hamas versus Palestine), not to the model's reasoning. Checking for systematic, explainable error clusters before scoring is a useful habit.
- They suggest labeling separately for each voice in a text, so that the reporter's stance is kept apart from the stance of quoted sources.
- A fixed definition of which surface names refer to each target entity, together with a glossary of terms marked as positive, neutral or negative, makes judgments repeatable.
- One of their diagnostic tests asks whether the text could be rewritten more neutrally, which treats bias as the gap between the text and a neutral counterpart.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Silver-data disagreements between witness parsers and an LLM teacher could be routed to review in the same way, instead of being resolved by majority vote alone.
- **I2** (learn): Teacher-tier outputs could be audited for repeated error patterns, and the missing constraint added to the prompt or to a repair rule.
- **I3** (frames): Speech-act and stance frames could carry a source attribute, so that a quoted claim is never attributed to the narrator.
- **I4** (world): Entity resolution could map aliases to canonical referents in the knowledge graph and attach connotation values to lexical items.
- **I5** (back): The generator could produce a neutral paraphrase from the meaning graph and compare it with the source to locate loaded wording.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [ZHAW Zurich University of Applied Sciences](https://md.research.elamur.ai/free/en/universities/zhaw-zurich-university-of-applied-sciences/) — University, Winterthur, Switzerland · 6 works · 181 citations
- [Università Cattolica del Sacro Cuore](https://md.research.elamur.ai/free/en/universities/universita-cattolica-del-sacro-cuore/) — University, Milan, Italy · 7 works · 33 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 577 works

## Cite

Cite as: Silvia Pareti, Jasmin Heierli, Serena Pareti and Tatiana Lando. 2024. Bias Bluff Busters at FIGNEWS 2024 Shared Task: Developing Guidelines to Make Bias Conscious. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{pareti2024bias,
  title = {{Bias Bluff Busters at FIGNEWS 2024 Shared Task: Developing Guidelines to Make Bias Conscious}},
  author = {Silvia Pareti, Jasmin Heierli, Serena Pareti and Tatiana Lando},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.62},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.62}
}
```

- [← previous BFCI at AraFinNLP2024: Support Vector Machines for Arabic Financial Text Classification](https://md.research.elamur.ai/free/en/papers/ashraf-2024-bfci-arafinnlp2024-support-vector-machines/)
- [next → Bibliography of Ethiopian Semitic, Cushitic, and Omotic Linguistics XXVI: 2022](https://md.research.elamur.ai/free/en/papers/krzyzanowska-2024-bibliography-ethiopian-semitic-cushitic-omotic/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/13/)

## References

9 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5636–5646, Hong Kong, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Giovanni+Da+San+Martino%2C+Seunghak+Yu%2C+Alberto+Barr%C3%B3n%2DCede%C3%B1o%2C+Rostislav+Petrov%2C+and+Preslav+Nakov%2E+2019%2E+Fine%2Dgrained+analysis+of+propaganda+in+news+article%2E+In+Proceedings+of+the+2019+Conference+on+Em)
2. Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. [Linguistic models for analyzing and detecting biased language](https://md.research.elamur.ai/free/en/papers/recasens-2013-linguistic-models-analyzing-detecting-biased/). In Proceedings of the 51st annual meeting of the Association for Computational Linguistics (volume 1: long papers), pages 1650–1659.
3. Timo Spinde. 2021. An interdisciplinary approach for the automated detection and visualization of media bias in news articles. In 2021 International Conference on Data Mining Workshops (ICDMW), pages 1096–1103. [find](https://scholar.google.com/scholar?q=Timo+Spinde%2E+2021%2E+An+interdisciplinary+approach+for+the+automated+detection+and+visualization+of+media+bias+in+news+articles%2E+In+2021+International+Conference+on+Data+Mining+Workshops+%28ICDMW%29%2C+pages+)
4. Timo Spinde, Smi Hinterreiter, Fabian Haak, Terry Ruas, Helge Giese, Norman Meuschke, and Bela Gipp. 2024. The media bias taxonomy: A systematic literature review on the forms and automated detection of media bias. Preprint, arXiv:2312.16148. [↗](https://arxiv.org/abs/2312.16148)
5. Wajdi Zaghouani, Mustafa Jarrar, Nizar Habash, Houda Bouamor, Imed Zitouni, Mona Diab, Samhaa R. El-Beltagy, and Muhammed AbuOdeh, editors. 2024. [The FIGNEWS Shared Task on News Media Narratives](https://md.research.elamur.ai/free/en/papers/zaghouani-2024-fignews-shared-task-news-media/). Association for Computational Linguistics, Bangkok, Thailand.
6. Michael Färber, Victoria Burkard, Adam Jatowt, and Sora Lim. 2020. A multidimensional dataset based on crowdsourcing for analyzing and detecting news bias. In Proceedings of CIKM 2020, CIKM ’20, page 3007–3014, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Michael+F%C3%A4rber%2C+Victoria+Burkard%2C+Adam+Jatowt%2C+and+Sora+Lim%2E+2020%2E+A+multidimensional+dataset+based+on+crowdsourcing+for+analyzing+and+detecting+news+bias%2E+In+Proceedings+of+CIKM+2020%2C+CIKM+%E2%80%9920%2C+page+)
7. Christoph Hube and Besnik Fetahu. 2019. Neural based statement classification for biased language. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, WSDM ’19, page 195–203, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Christoph+Hube+and+Besnik+Fetahu%2E+2019%2E+Neural+based+statement+classification+for+biased+language%2E+In+Proceedings+of+the+Twelfth+ACM+International+Conference+on+Web+Search+and+Data+Mining%2C+WSDM+%E2%80%9919%2C+p)
8. Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019. [Semeval-2019 task 4: Hyperpartisan news detection](https://md.research.elamur.ai/free/en/papers/kiesel-2019-semeval-2019-task-4-hyperpartisan/). In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 829–839.
9. Jakub Piskorski, Nicolas Stefanovitch, Valerie-Anne Bausier, Nicolo Faggiani, Jens Linge, Sopho Kharazi, Nikolaos Nikolaidis, Giulia Teodori, Bertrand De Longueville, Brian Doherty, et al. 2023. News categorization, framing and persuasion techniques: Annotation guidelines. Technical report, Technical report, European Commission Joint Research Centre, Ispra (Italy). [https://github.com/zhaw-iwi/FIGNEWS-2024](https://github.com/zhaw-iwi/FIGNEWS-2024) [↗](https://github.com/zhaw-iwi/FIGNEWS-2024)

*Updated 31 October 2026*