--- title: "Council of LLMs — paper by Sharma et al. (2026)" description: "This study asks whether large language models can reliably annotate propaganda, a subjective task where human… 0.69 F1. 1 reference." authors: ["Vivek Sharma", "Shweta Jain", "Mohammad Shokri", "Sarah Ita Levitan", "Elena Filatova"] year: 2026 venue: "The Proceedings for the 15th Workshop on Computational Approaches to Subjectivity, Sentiment Social Media Analysis (WASSA 2026)" source: "https://aclanthology.org/2026.wassa-1.1/" doi: "10.18653/v1/2026.wassa-1.1" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/sharma-2026-council-llms-evaluating-capability-large/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sharma-2026-council-llms-evaluating-capability-large/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/26/) / Council of LLMs: Evaluating Capability of Large Language Models to Annotate Propaganda # Council of LLMs: Evaluating Capability of Large Language Models to Annotate Propaganda **Authors:** [Vivek Sharma](https://md.research.elamur.ai/free/en/authors/vivek-sharma/), [Shweta Jain](https://md.research.elamur.ai/free/en/authors/shweta-jain/), [Mohammad Shokri](https://md.research.elamur.ai/free/en/authors/mohammad-shokri/), [Sarah Ita Levitan](https://md.research.elamur.ai/free/en/authors/sarah-ita-levitan/) and [Elena Filatova](https://md.research.elamur.ai/free/en/authors/elena-filatova/) · **Year:** 2026 · **Venue:** [The Proceedings for the 15th Workshop on Computational Approaches to Subjectivity, Sentiment Social Media Analysis (WASSA 2026)](https://md.research.elamur.ai/free/en/venues/wassa/) ([Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis 2026](https://md.research.elamur.ai/free/en/venues/wassa/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.wassa-1.1/) · [PDF](https://aclanthology.org/2026.wassa-1.1.pdf) · [DOI 10.18653/v1/2026.wassa-1.1](https://doi.org/10.18653/v1/2026.wassa-1.1) · [OpenAlex](https://openalex.org/W7140082637) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary This study asks whether large language models can reliably annotate propaganda, a subjective task where human labels are expensive and inconsistent. The authors compare three annotation setups. In the single-annotator setup, one model labels each sentence. In the multi-annotator setup, models label independently and then each model re-annotates after anonymously seeing all candidate labels and rationales. In the hybrid setup, human and LLM labels are combined. They test proprietary GPT models and open-source models on balanced 200-sentence samples from a news corpus (PTC) and a social media corpus (JMBX), averaging three runs per setting. No LLM configuration reached human-level accuracy. Hybrid annotation consolidated by GPT-4o came closest to the expert labels, followed by single LLM annotators. Multi-annotator consolidation performed worst, although it helped reasoning models. The models tend to over-detect propaganda, producing high recall but many false positives. A qualitative typology of their explanations shows that some disagreements with the gold labels are defensible interpretations rather than errors. The work supports using LLMs as aids within human-supervised annotation pipelines rather than as unsupervised labelers. ## Key points - **Data:** Propaganda Techniques Corpus (PTC; news, sentence-level labels) and Journalist Media Bias on X (JMBX; tweets from journalists at biased outlets). There were 200 sentences per dataset, split 50/50 between propaganda and non-propaganda, with results averaged over three runs. - **Models:** GPT-4o, GPT-4.1 and GPT-o3, plus open-source models including Llama 3.3 70B, QwQ, Qwen and DeepSeek R1. Prompts used the technique definitions of Martino et al. (2020). - **GPT results (P/R/F):** GPT-4.1 scored 0.77/0.74/0.74 standalone and 0.78/0.71/0.70 as consolidator. GPT-4o dropped from 0.79/0.74/0.73 to 0.78/0.70/0.68. Only the reasoning model GPT-o3 gained slightly (0.68 → 0.69 F1). Among open-source models, QwQ was the one that benefited from consolidation. - **Frequent false-positive labels:** Loaded Language was applied excessively more often than any other label, with Name Calling and Doubt next in line. Open-source models also sometimes assigned Causal Oversimplification to neutral text. - **Error typology of LLM reasoning:** the authors sorted disagreements into three types: - overgeneralization, where a definition is matched on the surface but without real persuasive intent; - defensible divergence, where the LLM's reading is plausible; - unconvincing or unsupported reasoning, based on shallow lexical cues. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Hunter College](https://md.research.elamur.ai/free/en/universities/hunter-college/) — University, New York, United States · 9 works · 1,245 citations - [The Graduate Center, CUNY](https://md.research.elamur.ai/free/en/universities/the-graduate-center-cuny/) — University, New York, United States · 13 works · 207 citations - [John Jay College of Criminal Justice](https://md.research.elamur.ai/free/en/universities/john-jay-college-of-criminal-justice/) — University, New York, United States · 2 works - [New York City College of Technology](https://md.research.elamur.ai/free/en/universities/new-york-city-college-of-technology/) — University, Brooklyn, United States · 2 works 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 577 works ## Cite Cite as: Vivek Sharma, Shweta Jain, Mohammad Shokri, Sarah Ita Levitan and Elena Filatova. 2026. Council of LLMs: Evaluating Capability of Large Language Models to Annotate Propaganda. The Proceedings for the 15th Workshop on Computational Approaches to Subjectivity, Sentiment Social Media Analysis (WASSA 2026). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{sharma2026council, title = {{Council of LLMs: Evaluating Capability of Large Language Models to Annotate Propaganda}}, author = {Vivek Sharma, Shweta Jain, Mohammad Shokri, Sarah Ita Levitan and Elena Filatova}, booktitle = {The Proceedings for the 15th Workshop on Computational Approaches to Subjectivity, Sentiment Social Media Analysis (WASSA 2026)}, year = {2026}, doi = {10.18653/v1/2026.wassa-1.1}, url = {https://doi.org/10.18653/v1/2026.wassa-1.1} } ``` - [← previous CoTrust: Privacy-Preserving Collaboration Between Large and Small Language Models in Trusted Execution Environments](https://md.research.elamur.ai/free/en/papers/ma-2026-cotrust-privacy-preserving-collaboration-between/) - [next → Counterspeech Generation using Small Language Models](https://md.research.elamur.ai/free/en/papers/muhammad-2026-counterspeech-generation-small-language-models/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/26/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. Abubakar Abid, Maheen Farooqi, and James Zou. 2021. [Persistent anti-muslim bias in large language models](https://md.research.elamur.ai/free/en/papers/abid-2021-persistent-anti-muslim-bias-large/). In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 298–306. \#\# Limitations *Updated 1 November 2026*