---
title: "Fine-Grained Analysis of Propaganda in News — Martino 2019"
description: "Most earlier propaganda detection work labels whole articles, often by copying the label of the news outlet that published them."
authors: ["Giovanni Da San Martino", "Seunghak Yu", "Alberto Barrón-Cedeño", "Rostislav Petrov", "Preslav Nakov"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1565/"
doi: "10.18653/v1/d19-1565"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/martino-2019-fine-grained-analysis-propaganda-news/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/martino-2019-fine-grained-analysis-propaganda-news/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/) / Fine-Grained Analysis of Propaganda in News Articles

# Fine-Grained Analysis of Propaganda in News Articles

**Authors:** [Giovanni Da San Martino](https://md.research.elamur.ai/free/en/authors/giovanni-da-san-martino/), Seunghak Yu, [Alberto Barrón-Cedeño](https://md.research.elamur.ai/free/en/authors/alberto-barron-cedeno/), [Rostislav Petrov](https://md.research.elamur.ai/free/en/authors/rostislav-petrov/), [Preslav Nakov](https://md.research.elamur.ai/free/en/authors/preslav-nakov/) · **Year:** 2019 · **Venue:** Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1565/) · [PDF](https://aclanthology.org/D19-1565.pdf) · [DOI 10.18653/v1/d19-1565](https://doi.org/10.18653/v1/d19-1565) · [OpenAlex](https://openalex.org/W2970487286)

**Citations:** 284 citations (openalex-snapshot · 2026-10-14) · [95](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Most earlier propaganda detection work labels whole articles, often by copying the label of the news outlet that published them. This yields noisy gold data, because biased outlets sometimes publish neutral pieces and reputable ones sometimes push an agenda. Such labels also cannot explain a prediction. The authors instead propose finding the exact text spans where a propaganda technique is used and naming the technique.

They define 18 techniques that can be judged from the text alone. Six professional annotators then labelled them in news articles from both propagandistic and non-propagandistic outlets, producing a corpus of about 350K tokens. The authors also design an evaluation measure suited to comparing partially overlapping labelled fragments.

For modelling, they introduce a multi-granularity neural network on top of BERT, in which sentence-level and token-level predictions inform each other. It beats several BERT-based baselines on both fragment-level (FLC) and sentence-level (SLC) detection. The corpus, measure and code are public, which makes this a foundation for explainable propaganda analysis.

## Key points
- The task inventory has 18 techniques, including loaded language, name calling, whataboutism, straw man and reductio ad hitlerum. Techniques that need external evidence to judge, such as card stacking, were deliberately excluded.
- The corpus is roughly 350K tokens annotated by six experts. It is released together with the evaluation code and models at propaganda.qcri.org.
- On SLC, the multi-granularity network with a ReLU gate reaches F1 60.98, compared with 57.74 for plain BERT. That is a gain of 8.42 recall points and 3.24 F1 points. A trivial baseline that labels every sentence as propaganda gets F1 38.61.
- On FLC, joint multi-task BERT performed worse than single-task BERT. The multi-granularity models came out best, mainly through higher precision, and there the Sigmoid gate beat ReLU.
- Pilot crowdsourcing showed that ordinary annotators rarely flag propaganda that matches their own views. This motivated the use of trained professionals.

## Interesting ideas
- **Gating fine-grained predictions with coarse ones:** the sentence-level output acts as a gate that suppresses token-level predictions in sentences judged clean. This could help any sparse span-detection task, such as hate speech spans or claim extraction, where most sentences contain nothing to find.
- **Picking the gate to match the weaker signal:** a soft Sigmoid gate worked better when the gating signal was noisy, while a harsher ReLU gate paid off in the other direction. This is a practical reminder to tune how strongly one noisy auxiliary prediction is allowed to override another.
- **Choosing labels that can be judged from the text alone:** keeping only techniques that are decidable without outside sources keeps annotation consistent and the task self-contained. This design rule is useful for disinformation datasets in any language, including Ukrainian media.
- **Expert annotation over the crowd for ideologically charged labels:** the paper's observation about annotator bias argues for trained annotators whenever a label depends on worldview, not just on language skill.
- **Collecting annotations through training:** the planned platform would let journalism students practise spotting techniques, with their annotations collected as a by-product. This is a cheap, sustainable way to grow expert-level data.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

95 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 92 not yet.

- [Transforming Causal LLM into MLM Encoder for Detecting Social Media Manipulation in Telegram](https://md.research.elamur.ai/free/en/papers/bazdyrev-2025-transforming-causal-llm-into-mlm/)
- [SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles](https://md.research.elamur.ai/free/en/papers/martino-2020-semeval-2020-task-11-detection/)
- [A Corpus of Persuasion Techniques in Slavic Languages](https://md.research.elamur.ai/free/en/papers/piskorski-2026-corpus-persuasion-techniques-slavic-languages/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{martino2019fine,
  title = {{Fine-Grained Analysis of Propaganda in News Articles}},
  author = {Martino, Giovanni Da San and Yu, Seunghak and Barrón-Cedeño, Alberto and Petrov, Rostislav and Nakov, Preslav},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1565},
  url = {https://doi.org/10.18653/v1/d19-1565}
}
```

- [← previous Findings of the WMT 2019 Shared Tasks on Quality Estimation](https://md.research.elamur.ai/free/en/papers/fonseca-2019-findings-wmt-2019-shared-tasks/)
- [next → FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP](https://md.research.elamur.ai/free/en/papers/akbik-2019-flair-easy-use-framework-state/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/)

## References

27 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alberto Barrón-Cedeño, Giovanni Da San Martino, Israa Jaradat, and Preslav Nakov. 2019. Proppy: A system to unmask propaganda in online news. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, AAAI ’19, pages 9847–9848, Honolulu, HI, USA. [find](https://scholar.google.com/scholar?q=Alberto+Barro%CC%81n%2DCeden%CC%83o%2C+Giovanni+Da+San+Martino%2C+Israa+Jaradat%2C+and+Preslav+Nakov%2E+2019%2E+Proppy%3A+A+system+to+unmask+propaganda+in+online+news%2E+In+Proceedings+of+the+33rd+AAAI+Conference+on+Artificial)
2. Alberto Barrón-Cedeno, Israa Jaradat, Giovanni Da San Martino, and Preslav Nakov. 2019. Proppy: Organizing the news based on their propagandistic content. Information Processing & Management, 56(5):1849–1864. [find](https://scholar.google.com/scholar?q=Alberto+Barro%CC%81n%2DCedeno%2C+Israa+Jaradat%2C+Giovanni+Da+San+Martino%2C+and+Preslav+Nakov%2E+2019%2E+Proppy%3A+Organizing+the+news+based+on+their+propagandistic+content%2E+Information+Processing+%26+Management%2C+56%285%29%3A1)
3. Lavinia Dan. 2015. Techniques for the Translation of Advertising Slogans. In Proceedings of the International Conference Literature, Discourse and Multicultural Dialogue, LDMD ’15, pages 13–23, Mures, Romania. [find](https://scholar.google.com/scholar?q=Lavinia+Dan%2E+2015%2E+Techniques+for+the+Translation+of+Advertising+Slogans%2E+In+Proceedings+of+the+International+Conference+Literature%2C+Discourse+and+Multicultural+Dialogue%2C+LDMD+%E2%80%9915%2C+pages+13%E2%80%9323%2C+Mures%2C)
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
5. Jean Goodwin. 2011. Accounting for the force of the appeal to authority. In Proceedings of the 9th International Conference of the Ontario Society for the Study of Argumentation, OSSA ’11, pages 1–9, Ontario, Canada. [find](https://scholar.google.com/scholar?q=Jean+Goodwin%2E+2011%2E+Accounting+for+the+force+of+the+appeal+to+authority%2E+In+Proceedings+of+the+9th+International+Conference+of+the+Ontario+Society+for+the+Study+of+Argumentation%2C+OSSA+%E2%80%9911%2C+pages+1%E2%80%939%2C+)
6. Ivan Habernal, Raffael Hannemann, Christian Pollak, Christopher Klamm, Patrick Pauli, and Iryna Gurevych. 2017. Argotario: Computational argumentation meets serious games. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’17, pages 7–12, Copenhagen, Denmark… [find](https://scholar.google.com/scholar?q=Ivan+Habernal%2C+Raffael+Hannemann%2C+Christian+Pollak%2C+Christopher+Klamm%2C+Patrick+Pauli%2C+and+Iryna+Gurevych%2E+2017%2E+Argotario%3A+Computational+argumentation+meets+serious+games%2E+In+Proceedings+of+the+Confer)
7. Ivan Habernal, Patrick Pauli, and Iryna Gurevych. 2018a. Adapting serious game for fallacious argumentation to German: pitfalls, insights, and best practices. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC ’18, Miyazaki, Japan. [find](https://scholar.google.com/scholar?q=Ivan+Habernal%2C+Patrick+Pauli%2C+and+Iryna+Gurevych%2E+2018a%2E+Adapting+serious+game+for+fallacious+argumentation+to+German%3A+pitfalls%2C+insights%2C+and+best+practices%2E+In+Proceedings+of+the+Eleventh+Internatio)
8. Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018b. Before name-calling: Dynamics and triggers of ad hominem fallacies in web argumentation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech… [find](https://scholar.google.com/scholar?q=Ivan+Habernal%2C+Henning+Wachsmuth%2C+Iryna+Gurevych%2C+and+Benno+Stein%2E+2018b%2E+Before+name%2Dcalling%3A+Dynamics+and+triggers+of+ad+hominem+fallacies+in+web+argumentation%2E+In+Proceedings+of+the+2018+Conference)
9. Renee Hobbs and Sandra Mcgee. 2008. Teaching about propaganda: An examination of the historical roots of media literacy. Journal of Media Literacy Education, 6(62):56–67. [find](https://scholar.google.com/scholar?q=Renee+Hobbs+and+Sandra+Mcgee%2E+2008%2E+Teaching+about+propaganda%3A+An+examination+of+the+historical+roots+of+media+literacy%2E+Journal+of+Media+Literacy+Education%2C+6%2862%29%3A56%E2%80%9367%2E)
10. Benjamin D Horne, Sara Khedr, and Sibel Adali. 2018. Sampling the news producers: A large news and feature data set for the study of the complex media landscape. In International AAAI Conference on Web and Social Media, ICWSM ’18, Stanford, CA, USA. [find](https://scholar.google.com/scholar?q=Benjamin+D+Horne%2C+Sara+Khedr%2C+and+Sibel+Adali%2E+2018%2E+Sampling+the+news+producers%3A+A+large+news+and+feature+data+set+for+the+study+of+the+complex+media+landscape%2E+In+International+AAAI+Conference+on+We)
11. John Hunter. 2015. Brainwashing in a large group awareness training? The classical conditioning hypothesis of brainwashing. Master’s thesis, University of Kwazulu-Natal, Pietermaritzburg, South Africa. [find](https://scholar.google.com/scholar?q=John+Hunter%2E+2015%2E+Brainwashing+in+a+large+group+awareness+training%3F+The+classical+conditioning+hypothesis+of+brainwashing%2E+Master%E2%80%99s+thesis%2C+University+of+Kwazulu%2DNatal%2C+Pietermaritzburg%2C+South+Africa)
12. Garth S. Jowett and Victoria O’Donnell. 2012. What is propaganda, and how does it differ from persuasion? In Propaganda & Persuasion, chapter 1, pages 1–48. Sage Publishing. [find](https://scholar.google.com/scholar?q=Garth+S%2E+Jowett+and+Victoria+O%E2%80%99Donnell%2E+2012%2E+What+is+propaganda%2C+and+how+does+it+differ+from+persuasion%3F+In+Propaganda+%26+Persuasion%2C+chapter+1%2C+pages+1%E2%80%9348%2E+Sage+Publishing%2E)
13. Yann Mathet, Antoine Widlöcher, and Jean-Philippe Métivier. 2015. The unified and holistic method gamma (γ) for inter-annotator agreement measure and alignment. Computational Linguistics, 41(3):437–479. [find](https://scholar.google.com/scholar?q=Yann+Mathet%2C+Antoine+Widlo%CC%88cher%2C+and+Jean%2DPhilippe+Me%CC%81tivier%2E+2015%2E+The+unified+and+holistic+method+gamma+%28%CE%B3%29+for+inter%2Dannotator+agreement+measure+and+alignment%2E+Computational+Linguistics%2C+41%283%29%3A437%E2%80%93)
14. Christian M. Meyer, Margot Mieskes, Christian Stab, and Iryna Gurevych. 2014. DKPro agreement: An open-source Java library for measuring inter-rater agreement. In Proceedings of the International Conference on Computational Linguistics, COL-ING ’14, pages 105–109, Dublin, Ireland. [find](https://scholar.google.com/scholar?q=Christian+M%2E+Meyer%2C+Margot+Mieskes%2C+Christian+Stab%2C+and+Iryna+Gurevych%2E+2014%2E+DKPro+agreement%3A+An+open%2Dsource+Java+library+for+measuring+inter%2Drater+agreement%2E+In+Proceedings+of+the+International+Conf)
15. Clyde R. Miller. 1939. The Techniques of Propaganda. From “How to Detect and Analyze Propaganda,” an address given at Town Hall. The Center for learning. [find](https://scholar.google.com/scholar?q=Clyde+R%2E+Miller%2E+1939%2E+The+Techniques+of+Propaganda%2E+From+%E2%80%9CHow+to+Detect+and+Analyze+Propaganda%2C%E2%80%9D+an+address+given+at+Town+Hall%2E+The+Center+for+learning%2E)
16. David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. Lingvisticae Investigationes, 30(1):3–26. [find](https://scholar.google.com/scholar?q=David+Nadeau+and+Satoshi+Sekine%2E+2007%2E+A+survey+of+named+entity+recognition+and+classification%2E+Lingvisticae+Investigationes%2C+30%281%29%3A3%E2%80%9326%2E)
17. Martin Potthast, Benno Stein, Alberto Barrón-Cedeño, and Paolo Rosso. 2010. An evaluation framework for plagiarism detection. In Proceedings of the International Conference on Computational Linguistics, COLING ’10, pages 997–1005, Beijing, China. [find](https://scholar.google.com/scholar?q=Martin+Potthast%2C+Benno+Stein%2C+Alberto+Barro%CC%81n%2DCeden%CC%83o%2C+and+Paolo+Rosso%2E+2010%2E+An+evaluation+framework+for+plagiarism+detection%2E+In+Proceedings+of+the+International+Conference+on+Computational+Linguist)
18. Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP ’17, pages 2931–2937, Copenhagen,… [find](https://scholar.google.com/scholar?q=Hannah+Rashkin%2C+Eunsol+Choi%2C+Jin+Yea+Jang%2C+Svitlana+Volkova%2C+and+Yejin+Choi%2E+2017%2E+Truth+of+varying+shades%3A+Analyzing+language+in+fake+news+and+political+fact%2Dchecking%2E+In+Proceedings+of+the+2017+Conf) [our card](https://md.research.elamur.ai/free/en/papers/rashkin-2017-truth-varying-shades-analyzing-language/)
19. Monika L Richter. 2017. The Kremlin’s platform for ‘useful idiots’ in the West: An overview of RT’s editorial strategy and evidence of impact. Technical report, Kremlin Watch. [find](https://scholar.google.com/scholar?q=Monika+L+Richter%2E+2017%2E+The+Kremlin%E2%80%99s+platform+for+%E2%80%98useful+idiots%E2%80%99+in+the+West%3A+An+overview+of+RT%E2%80%99s+editorial+strategy+and+evidence+of+impact%2E+Technical+report%2C+Kremlin+Watch%2E)
20. Francisca Niken Vitri Suprabandari. 2007. American propaganda in John Steinbeck’s The Moon is Down. Master’s thesis, Sanata Dharma University, Yogyakarta, Indonesia. [find](https://scholar.google.com/scholar?q=Francisca+Niken+Vitri+Suprabandari%2E+2007%2E+American+propaganda+in+John+Steinbeck%E2%80%99s+The+Moon+is+Down%2E+Master%E2%80%99s+thesis%2C+Sanata+Dharma+University%2C+Yogyakarta%2C+Indonesia%2E)
21. Gabriel H Teninbaum. 2009. Reductio ad Hitlerum: Trumping the judicial Nazi card. Michigan State Law Review, page 541. [find](https://scholar.google.com/scholar?q=Gabriel+H+Teninbaum%2E+2009%2E+Reductio+ad+Hitlerum%3A+Trumping+the+judicial+Nazi+card%2E+Michigan+State+Law+Review%2C+page+541%2E)
22. Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In Proceedings of the 6th Conference on Natural Language Learning, CoNLL ’02, pages 155–158, Taipei, Taiwan. [find](https://scholar.google.com/scholar?q=Erik+F%2E+Tjong+Kim+Sang%2E+2002%2E+Introduction+to+the+CoNLL%2D2002+shared+task%3A+Language%2Dindependent+named+entity+recognition%2E+In+Proceedings+of+the+6th+Conference+on+Natural+Language+Learning%2C+CoNLL+%E2%80%9902%2C+p) [our card](https://md.research.elamur.ai/free/en/papers/sang-2002-introduction-conll-2002-shared-task/)
23. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the 7th Conference on Natural Language Learning, CoNLL ’03, pages 142–147, Edmonton, Canada. [find](https://scholar.google.com/scholar?q=Erik+F%2E+Tjong+Kim+Sang+and+Fien+De+Meulder%2E+2003%2E+Introduction+to+the+CoNLL%2D2003+shared+task%3A+Language%2Dindependent+named+entity+recognition%2E+In+Proceedings+of+the+7th+Conference+on+Natural+Language+Le)
24. Robyn Torok. 2015. Symbiotic radicalisation strategies: Propaganda tools and neuro linguistic programming. In Proceedings of the Australian Security and Intelligence Conference, pages 58–65, Perth, Australia. [find](https://scholar.google.com/scholar?q=Robyn+Torok%2E+2015%2E+Symbiotic+radicalisation+strategies%3A+Propaganda+tools+and+neuro+linguistic+programming%2E+In+Proceedings+of+the+Australian+Security+and+Intelligence+Conference%2C+pages+58%E2%80%9365%2C+Perth%2C+Au)
25. Richard Tzong-Han Tsai, Shih-Hung Wu, Wen-Chi Chou, Yu-Chun Lin, Ding He, Jieh Hsiang, Ting-Yi Sung, and Wen-Lian Hsu. 2006. Various criteria in the evaluation of biomedical named entity recognition. BMC bioinformatics, 7:92. [find](https://scholar.google.com/scholar?q=Richard+Tzong%2DHan+Tsai%2C+Shih%2DHung+Wu%2C+Wen%2DChi+Chou%2C+Yu%2DChun+Lin%2C+Ding+He%2C+Jieh+Hsiang%2C+Ting%2DYi+Sung%2C+and+Wen%2DLian+Hsu%2E+2006%2E+Various+criteria+in+the+evaluation+of+biomedical+named+entity+recognition%2E+)
26. Douglas Walton. 1996. The straw man fallacy. Royal Netherlands Academy of Arts and Sciences. [find](https://scholar.google.com/scholar?q=Douglas+Walton%2E+1996%2E+The+straw+man+fallacy%2E+Royal+Netherlands+Academy+of+Arts+and+Sciences%2E)
27. Anthony Weston. 2018. A rulebook for arguments. Hackett Publishing. [find](https://scholar.google.com/scholar?q=Anthony+Weston%2E+2018%2E+A+rulebook+for+arguments%2E+Hackett+Publishing%2E)