---
title: "DRAGON at FIGNEWS 2024 Shared Task: a Dedicated RAG for"
description: "The paper describes a team entry to the FIGNEWS 2024 shared task, which asks… 35.7 accuracy. 18 references, cited by 1 paper here."
authors: ["Sadegh Jafari", "Mohsen Mahmoodzadeh", "Vanooshe Nazari", "Razieh Bahmanyar", "Kathryn Burrows"]
year: 2024
venue: "Proceedings of the Second Arabic Natural Language Processing Conference"
source: "https://aclanthology.org/2024.arabicnlp-1.58/"
doi: "10.18653/v1/2024.arabicnlp-1.58"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jafari-2024-dragon-fignews-2024-shared-task/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jafari-2024-dragon-fignews-2024-shared-task/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/13/) / DRAGON at FIGNEWS 2024 Shared Task: a Dedicated RAG for October 7th conflict News

# DRAGON at FIGNEWS 2024 Shared Task: a Dedicated RAG for October 7th conflict News

**Authors:** Sadegh Jafari, Mohsen Mahmoodzadeh, Vanooshe Nazari, Razieh Bahmanyar and Kathryn Burrows · **Year:** 2024 · **Venue:** [Proceedings of the Second Arabic Natural Language Processing Conference](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2024](https://md.research.elamur.ai/free/en/venues/arabicnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.arabicnlp-1.58/) · [PDF](https://aclanthology.org/2024.arabicnlp-1.58.pdf) · [DOI 10.18653/v1/2024.arabicnlp-1.58](https://doi.org/10.18653/v1/2024.arabicnlp-1.58) · [OpenAlex](https://openalex.org/W4402671043)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/jafari-2024-dragon-fignews-2024-shared-task/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper describes a team entry to the FIGNEWS 2024 shared task, which asks participants to annotate news and social-media posts about the Gaza–Israel war of 2023–2024 for bias and propaganda. Annotation budgets were small, so the authors first chose which posts to label. They clustered the 13,500-item dataset with BERTopic, using e5-base embeddings, and drew about 50 posts from each of 35 topics, giving 1,812 posts. Two human annotators, one from Iran and one from the USA, labelled this subset. They worked from written guidelines with 7 bias tags and 4 propaganda tags. The labelled subset was then used to train four "artificial annotators": KNN, SVC and XGBoost over sentence embeddings, plus a retrieval-augmented GPT-3.5-turbo classifier. A vote combines their outputs to label the remaining data. Bias classification reached weighted F1 of roughly 0.71–0.78, and propaganda roughly 0.55–0.63. The authors argue that topic-based sampling is a cheap way to get a representative seed set. They also note that label imbalance and the small, culturally split annotator pool limit reliability.

## Key points
- The data pipeline goes from 13,500 FIGNEWS samples to 35 BERTopic clusters (e5-base features) to a 1,812-sample annotated subset. The subset was split into train and test.
- The label distribution is very skewed:
  - "Unbiased" has 1,224 training examples.
  - "Biased against both Palestine and Israel" has only 6 for training and 2 for testing.
  - Propaganda labels are more even: 799 "Not Propaganda" and 427 "Propaganda" in training.
- Several embedders were tested as classifier inputs: Multilingual E5 large, BGE M3, E5 Mistral 7B, Nomic-Embed and XLM-Roberta. KNN (k=5) was reported as the strongest classic classifier.
- The RAG classifier works as follows:
  - It retrieves from a vector store built with Multilingual E5 large.
  - Its prompt combines the label definitions with similar annotated examples. The authors report that taking 5 nearest documents and keeping one example per distinct label worked well.
  - GPT-3.5-turbo then picks the label.
- Committee-reported results:
  - Within-team quality: Kappa 35.7, accuracy 75.5, macro F1 41.0.
  - Cross-team centrality: Kappa 19.7, accuracy 41.1, macro F1 21.9.

## Interesting ideas
- Picking the items to hand-label by sampling evenly across topic clusters, instead of at random, keeps rare themes in a small seed set.
- The few-shot prompt is built per input: it holds every label's definition plus the retrieved nearest annotated example for each distinct label, so the model sees one contrast case per option.
- Ensemble votes are broken by trusting the model with the higher held-out F1, which gives a simple, explainable rule when witnesses split evenly.
- Cases where a human and the model disagree are not just corrected. They are used to revise the written label definitions, so the guidelines change along with the data.
- The authors show that a label with only a handful of examples makes test scores unstable, and they recommend cross-validation over a single split.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): When choosing sentences for LLM-teacher or human review, cluster the corpus (e.g., by frame or verb class) and sample per cluster, so the silver data is not dominated by frequent constructions.
- **I2** (learn): LLM teacher prompts for ambiguous UD relations or VerbNet roles could include each candidate label's definition and one retrieved gold example per candidate.
- **I3** (learn): When witness parsers disagree evenly on an arc or label, defer to the parser with the best measured score for that relation type and log the tie-break.
- **I4** (frames): Disagreements between prag's frame or speech-act output and teacher labels could be logged as triggers for refining the frame definitions themselves.
- **I5** (answer): Evaluations on small question sets (fable QA, word problems) should report per-category counts and use cross-validation or repeated splits before drawing conclusions.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Iran University of Science and Technology](https://md.research.elamur.ai/free/en/universities/iran-university-of-science-and-technology/) — University, Tehran, Iran · 4 works · 65 citations

Also on the byline: [Ferdowsi University of Mashhad](https://ror.org/00g6ka752) · [Madonna University](https://ror.org/02rg9ze43) · [Madonna University](https://ror.org/01r5xd980).

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 577 works

💡 Concepts (1): [💡 Topic modeling](https://md.research.elamur.ai/free/en/concepts/topic-modeling/) · 47 works

## Cite

Cite as: Sadegh Jafari, Mohsen Mahmoodzadeh, Vanooshe Nazari, Razieh Bahmanyar and Kathryn Burrows. 2024. DRAGON at FIGNEWS 2024 Shared Task: a Dedicated RAG for October 7th conflict News. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{jafari2024dragon,
  title = {{DRAGON at FIGNEWS 2024 Shared Task: a Dedicated RAG for October 7th conflict News}},
  author = {Sadegh Jafari, Mohsen Mahmoodzadeh, Vanooshe Nazari, Razieh Bahmanyar and Kathryn Burrows},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.58},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.58}
}
```

- [← previous dkit@LT-EDI-2024: Detecting Homophobia and Transphobia in English Social Media Comments](https://md.research.elamur.ai/free/en/papers/yadav-2024-dkit-lt-edi-2024-detecting/)
- [next → Dreaming Out Loud: A Self-Synthesis Approach For Training Vision-Language Models With Developmentally Plausible Data](https://md.research.elamur.ai/free/en/papers/alkhamissi-2024-dreaming-out-loud-self-synthesis/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/13/)

## References

18 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Kylie Anglin, Arielle Boguslav, and Todd Hall. 2020. Improving the science of annotation for natural language processing. Grantee Submission. [find](https://scholar.google.com/scholar?q=Kylie+Anglin%2C+Arielle+Boguslav%2C+and+Todd+Hall%2E+2020%2E+Improving+the+science+of+annotation+for+natural+language+processing%2E+Grantee+Submission%2E)
2. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. [Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation](https://md.research.elamur.ai/free/en/papers/chen-2024-m3-embedding-multi-linguality/). Preprint, arXiv:2402.03216. [↗](https://arxiv.org/abs/2402.03216)
3. Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785– 794. [↗](https://doi.org/10.48550/arxiv.1603.02754)
4. Costanza Conforti, Stephanie Hirmer, David Morgan, Marco Basaldella, and Yau Ben Or. 2020. Natural language processing for achieving sustainable development: the case of neural labelling to enhance community profiling. arXiv preprint arXiv:2004.12935. [↗](https://arxiv.org/abs/2004.12935)
5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. [↗](https://arxiv.org/abs/1911.02116)
6. Thomas Cover and Peter Hart. 1967. Nearest neighbor pattern classification. IEEE transactions on information theory, 13(1):21–27. [↗](https://doi.org/10.1109/tit.1967.1053964)
7. Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Shafiq Joty, Boyang Li, and Lidong Bing. 2022. [Is gpt-3 a good data annotator?](https://md.research.elamur.ai/free/en/papers/ding-2022-gpt-3-good-data-annotator/) arXiv preprint arXiv:2212.10450. [↗](https://arxiv.org/abs/2212.10450)
8. Maarten Grootendorst. 2022. [Bertopic: Neural topic modeling with a class-based tf-idf procedure](https://md.research.elamur.ai/free/en/papers/grootendorst-2022-bertopic-neural-topic-modeling-class/). arXiv preprint arXiv:2203.05794. [↗](https://arxiv.org/abs/2203.05794)
9. Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. 1998. Support vector machines. IEEE Intelligent Systems and their applications, 13(4):18–28. [↗](https://doi.org/10.1109/5254.708428)
10. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. [What makes good in-context examples for gpt-3?](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/) arXiv preprint arXiv:2101.06804. [↗](https://arxiv.org/abs/2101.06804)
11. Zach Nussbaum, John X. Morris, Brandon Duderstadt, and Andriy Mulyar. 2024. [Nomic embed: Training a reproducible long context text embedder](https://md.research.elamur.ai/free/en/papers/nussbaum-2024-nomic-embed-training-reproducible-long/). Preprint, arXiv:2402.01613. [↗](https://arxiv.org/abs/2402.01613)
12. Tal Perry. 2021. Lighttag: Text annotation platform. arXiv preprint arXiv:2109.02320. [↗](https://arxiv.org/abs/2109.02320)
13. Marek Šuppa, Daniel Skala, Daniela Jašš, Samuel Sučík, Andrej Švec, and Peter Hraška. 2024. Bryndza at climateactivism 2024: Stance, target and hate event detection via retrieval-augmented gpt-4 and llama. arXiv preprint arXiv:2402.06549. [↗](https://arxiv.org/abs/2402.06549)
14. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023. [Improving text embeddings with large language models](https://md.research.elamur.ai/free/en/papers/wang-2024-improving-text-embeddings-large-language/). arXiv preprint arXiv:2401.00368. [↗](https://arxiv.org/abs/2401.00368)
15. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. [Multilingual e5 text embeddings: A technical report](https://md.research.elamur.ai/free/en/papers/wang-2024-multilingual-e5-text-embeddings-technical/). arXiv preprint arXiv:2402.05672. [↗](https://arxiv.org/abs/2402.05672)
16. Wajdi Zaghouani, Mustafa Jarrar, Nizar Habash, Houda Bouamor, Imed Zitouni, Mona Diab, Samhaa R. El-Beltagy, and Muhammed Raed AbuOdeh, editors. 2024. [The FIGNEWS Shared Task on News Media Narratives](https://md.research.elamur.ai/free/en/papers/zaghouani-2024-fignews-shared-task-news-media/). Association for Computational Linguistics, Bangkok, Thailand.
17. Boyu Zhang, Hongyang Yang, Tianyu Zhou, Muhammad Ali Babar, and Xiao-Yang Liu. 2023. Enhancing financial sentiment analysis via retrieval augmented large language models. In Proceedings of the Fourth ACM International Conference on AI in Finance, pages 349–356. [find](https://scholar.google.com/scholar?q=Boyu+Zhang%2C+Hongyang+Yang%2C+Tianyu+Zhou%2C+Muhammad+Ali+Babar%2C+and+Xiao%2DYang+Liu%2E+2023%2E+Enhancing+financial+sentiment+analysis+via+retrieval+augmented+large+language+models%2E+In+Proceedings+of+the+Fourth+)
18. Shikun Zhang, Omid Jafari, and Parth Nagarkar. 2021. A survey on machine learning techniques for auto labeling of video, audio, and text data. arXiv preprint arXiv:2109.03784. [↗](https://arxiv.org/abs/2109.03784)

*Updated 1 November 2026*