---
title: "Validating Automatic Evaluation of Controllable Counterspeech"
description: "Zheng, Ross and Magdy propose adding a classifier validation step between building a classifier and using it to compare models. 29 references."
authors: ["Yi Zheng", "Björn Ross", "Walid Magdy"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.eacl-long.193/"
doi: "10.18653/v1/2026.eacl-long.193"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zheng-2026-validating-automatic-evaluation-controllable-counterspeech/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zheng-2026-validating-automatic-evaluation-controllable-counterspeech/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/55/) / Validating Automatic Evaluation of Controllable Counterspeech Generation: Rankings Matter More Than Scores

# Validating Automatic Evaluation of Controllable Counterspeech Generation: Rankings Matter More Than Scores

**Authors:** [Yi Zheng](https://md.research.elamur.ai/free/en/authors/yi-zheng/), Björn Ross and [Walid Magdy](https://md.research.elamur.ai/free/en/authors/walid-magdy/) · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-long.193/) · [PDF](https://aclanthology.org/2026.eacl-long.193.pdf) · [DOI 10.18653/v1/2026.eacl-long.193](https://doi.org/10.18653/v1/2026.eacl-long.193) · [OpenAlex](https://openalex.org/W7140136978)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Studies on controllable counterspeech generation often judge whether outputs show a target attribute, such as a strategy or stance, by running automatic classifiers. These classifiers frequently have only modest F1, which raises doubts about using them to compare generation models. Zheng, Ross and Magdy propose adding a classifier validation step between building a classifier and using it to compare models. In this step, a classifier counts as trustworthy if its ranking of generation models agrees with human judgements, regardless of its accuracy or F1. To test this, they annotated a new English multi-attribute dataset and fine-tuned classifiers on all annotations together and on each annotator's labels separately. They then checked how well each classifier ranked six LLMs. Imperfect classifiers often still produced reliable model rankings. Disagreement between annotators usually had little effect on these rankings, with Hypocrisy as a notable exception. The authors conclude that ranking agreement with humans, not classification scores, should be the main criterion for choosing an evaluation classifier.

## Key points
- **Dataset:** 2,728 counterspeech replies to 880 hate speech items, taken from the Fanton et al. (2021) data. Five annotators (A1–A5) labelled four strategies (Factual, Denouncing, Empathy, Hypocrisy) and a Stance label (neutral, positive or antagonistic). The dataset is released publicly on OSF.
- **Annotation agreement:** Agreement was low to moderate, with final Fleiss' κ ranging from 0.280 for Hypocrisy to 0.438 for Factual.
- **Classifier performance:** The fine-tuned classifiers reached macro-F1 between 0.64 and 0.85.
- **Validation setup:** Six LLMs each generated 1,000 counterspeech samples zero-shot. Classifier rankings of these LLMs were compared with human judgements.
- **Annotator effects:**
  - For Empathy and Stance, classifiers trained on a single annotator's labels ranked models much like classifiers trained on all annotations.
  - For Hypocrisy, annotator-specific classifiers produced divergent rankings.
  - For Factual, the annotator-specific classifiers agreed with each other on rankings, but only C_A1 and C_A4 passed validation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Edinburgh](https://md.research.elamur.ai/free/en/universities/university-of-edinburgh/) — University, Edinburgh, United Kingdom · 381 works · 80,747 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 626 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works

## Cite

Cite as: Yi Zheng, Björn Ross and Walid Magdy. 2026. Validating Automatic Evaluation of Controllable Counterspeech Generation: Rankings Matter More Than Scores. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zheng2026validating,
  title = {{Validating Automatic Evaluation of Controllable Counterspeech Generation: Rankings Matter More Than Scores}},
  author = {Yi Zheng, Björn Ross and Walid Magdy},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.193},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.193}
}
```

- [← previous VAIDYA: Validated Agents for Intelligent Diagnosis and Yielded Analysis](https://md.research.elamur.ai/free/en/papers/shah-2026-vaidya-validated-agents-intelligent-diagnosis/)
- [next → Validator-Guided Hard Negative Mining for Masked Language Modeling in Low-Resource Ancient Languages](https://md.research.elamur.ai/free/en/papers/voinea-2026-validator-guided-hard-negative-mining/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/55/)

## References

29 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, and 1 others. 2024. [Phi-4 technical report](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-4-technical-report/). arXiv preprint arXiv:2412.08905.
2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
3. Susan Benesch, Derek Ruths, Kelly P Dillon, Haji Mohammad Saleem, and Lucas Wright. 2016. Considerations for successful counterspeech. Dangerous speech project. [find](https://scholar.google.com/scholar?q=Susan+Benesch%2C+Derek+Ruths%2C+Kelly+P+Dillon%2C+Haji+Mohammad+Saleem%2C+and+Lucas+Wright%2E+2016%2E+Considerations+for+successful+counterspeech%2E+Dangerous+speech+project%2E)
4. Yi-Ling Chung, Marco Guerini, and Rodrigo Agerri. 2021. Multilingual counter narrative type classification. In Proceedings of the 8th Workshop on Argument Mining, pages 125–132. [find](https://scholar.google.com/scholar?q=Yi%2DLing+Chung%2C+Marco+Guerini%2C+and+Rodrigo+Agerri%2E+2021%2E+Multilingual+counter+narrative+type+classification%2E+In+Proceedings+of+the+8th+Workshop+on+Argument+Mining%2C+pages+125%E2%80%93132%2E)
5. Yi-Ling Chung, Elizaveta Kuzmenko, Serra Sinem Tekiroğlu, and Marco Guerini. 2019. [Conan-counter narratives through nichesourcing: a multilingual dataset of responses to fight online hate speech](https://md.research.elamur.ai/free/en/papers/chung-2019-conan-counter-narratives-through-nichesourcing/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2819–…
6. Margherita Fanton, Helena Bonaldi, Serra Sinem Tekiroğlu, and Marco Guerini. 2021. [Human-in-the-loop for data collection: a multi-target counter narrative dataset to fight online hate speech](https://md.research.elamur.ai/free/en/papers/fanton-2021-human-loop-data-collection-multi/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Int…
7. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. [find](https://scholar.google.com/scholar?q=Joseph+L+Fleiss%2E+1971%2E+Measuring+nominal+scale+agreement+among+many+raters%2E+Psychological+bulletin%2C+76%285%29%3A378%2E)
8. Robert Gorwa, Reuben Binns, and Christian Katzenbach. 2020. Algorithmic content moderation: Technical and political challenges in the automation of platform governance. Big Data & Society, 7(1):2053951719897945. [find](https://scholar.google.com/scholar?q=Robert+Gorwa%2C+Reuben+Binns%2C+and+Christian+Katzenbach%2E+2020%2E+Algorithmic+content+moderation%3A+Technical+and+political+challenges+in+the+automation+of+platform+governance%2E+Big+Data+%26+Society%2C+7%281%29%3A205395)
9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
10. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
11. Rishabh Gupta, Shaily Desai, Manvi Goel, Anil Bandhakavi, Tanmoy Chakraborty, and Md Shad Akhtar. 2023. Counterspeeches up my sleeve! intent distribution learning and persistent fusion for intent-conditioned counterspeech generation. In The 61st Annual Meeting Of The Association For Computational Li… [find](https://scholar.google.com/scholar?q=Rishabh+Gupta%2C+Shaily+Desai%2C+Manvi+Goel%2C+Anil+Bandhakavi%2C+Tanmoy+Chakraborty%2C+and+Md+Shad+Akhtar%2E+2023%2E+Counterspeeches+up+my+sleeve%21+intent+distribution+learning+and+persistent+fusion+for+intent%2Dcond)
12. Jason S Haukoos and Roger J Lewis. 2005. Advanced statistics: bootstrapping confidence intervals for statistics with “difficult” distributions. Academic emergency medicine, 12(4):360–365. [find](https://scholar.google.com/scholar?q=Jason+S+Haukoos+and+Roger+J+Lewis%2E+2005%2E+Advanced+statistics%3A+bootstrapping+confidence+intervals+for+statistics+with+%E2%80%9Cdifficult%E2%80%9D+distributions%2E+Academic+emergency+medicine%2C+12%284%29%3A360%E2%80%93365%2E)
13. Amey Hengle, Aswini Padhi, Sahajpreet Singh, Anil Bandhakavi, Md Shad Akhtar, and Tanmoy Chakraborty. 2024. Intent-conditioned and non-toxic counterspeech generation using multi-task instruction tuning with rlaif. In Proceedings of the 2024 Conference of the North American Chapter of the Association… [find](https://scholar.google.com/scholar?q=Amey+Hengle%2C+Aswini+Padhi%2C+Sahajpreet+Singh%2C+Anil+Bandhakavi%2C+Md+Shad+Akhtar%2C+and+Tanmoy+Chakraborty%2E+2024%2E+Intent%2Dconditioned+and+non%2Dtoxic+counterspeech+generation+using+multi%2Dtask+instruction+tunin)
14. Sameer Hinduja. Generative ai as a vector for harassment and harm. [https://cyberbullying.org/generative-ai-as-a-vector-for-harassment-and-harm](https://cyberbullying.org/generative-ai-as-a-vector-for-harassment-and-harm). Accessed: 2025-04-10. [↗](https://cyberbullying.org/generative-ai-as-a-vector-for-harassment-and-harm)
15. Aswini Kumar, Anil Bandhakavi, and Tanmoy Chakraborty. 2025. Counterspeech the ultimate shield! multi-conditioned counterspeech generation through attributed prefix learning. arXiv preprint arXiv:2505.11958. [find](https://scholar.google.com/scholar?q=Aswini+Kumar%2C+Anil+Bandhakavi%2C+and+Tanmoy+Chakraborty%2E+2025%2E+Counterspeech+the+ultimate+shield%21+multi%2Dconditioned+counterspeech+generation+through+attributed+prefix+learning%2E+arXiv+preprint+arXiv%3A2505)
16. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+and+1+others%2E+2024%2E+Deepseek%2Dv3+technical+report%2E+arXiv+preprint+arXiv%3A2412)
17. Yangguang Liu, Yangming Zhou, Shiting Wen, and Chaogang Tang. 2014. A strategy on selecting performance metrics for classifier evaluation. International Journal of Mobile Computing and Multimedia Communications (IJMCMC), 6(4):20–35. [find](https://scholar.google.com/scholar?q=Yangguang+Liu%2C+Yangming+Zhou%2C+Shiting+Wen%2C+and+Chaogang+Tang%2E+2014%2E+A+strategy+on+selecting+performance+metrics+for+classifier+evaluation%2E+International+Journal+of+Mobile+Computing+and+Multimedia+Comm)
18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692.
19. Binny Mathew, Punyajoy Saha, Hardik Tharad, Subham Rajgaria, Prajwal Singhania, Suman Kalyan Maity, Pawan Goyal, and Animesh Mukherjee. 2019. Thou shalt not hate: Countering online hate speech. In Proceedings of the international AAAI conference on web and social media, volume 13, pages 369–380. [find](https://scholar.google.com/scholar?q=Binny+Mathew%2C+Punyajoy+Saha%2C+Hardik+Tharad%2C+Subham+Rajgaria%2C+Prajwal+Singhania%2C+Suman+Kalyan+Maity%2C+Pawan+Goyal%2C+and+Animesh+Mukherjee%2E+2019%2E+Thou+shalt+not+hate%3A+Countering+online+hate+speech%2E+In+Pro)
20. Jimin Mun, Emily Allaway, Akhila Yerukola, Laura Vianna, Sarah-Jane Leslie, and Maarten Sap. 2023. Beyond denouncing hate: Strategies for countering implied biases and stereotypes in language. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9759–9777. [find](https://scholar.google.com/scholar?q=Jimin+Mun%2C+Emily+Allaway%2C+Akhila+Yerukola%2C+Laura+Vianna%2C+Sarah%2DJane+Leslie%2C+and+Maarten+Sap%2E+2023%2E+Beyond+denouncing+hate%3A+Strategies+for+countering+implied+biases+and+stereotypes+in+language%2E+In+Find)
21. Antonio Nucci. 2025. Llm evaluation: Key metrics and frameworks. Accessed: 2025-Sep. [find](https://scholar.google.com/scholar?q=Antonio+Nucci%2E+2025%2E+Llm+evaluation%3A+Key+metrics+and+frameworks%2E+Accessed%3A+2025%2DSep%2E)
22. Aashima Poudhar, Ioannis Konstas, and Gavin Abercrombie. 2024. A strategy labelled dataset of counterspeech. In Proceedings of the 8th Workshop on Online Abuse and Harms (WOAH 2024), pages 256– 265. [find](https://scholar.google.com/scholar?q=Aashima+Poudhar%2C+Ioannis+Konstas%2C+and+Gavin+Abercrombie%2E+2024%2E+A+strategy+labelled+dataset+of+counterspeech%2E+In+Proceedings+of+the+8th+Workshop+on+Online+Abuse+and+Harms+%28WOAH+2024%29%2C+pages+256%E2%80%93+265%2E)
23. Jing Qian, Anna Bethke, Yinyin Liu, Elizabeth Belding, and William Yang Wang. 2019. [A benchmark dataset for learning to intervene in online hate speech](https://md.research.elamur.ai/free/en/papers/qian-2019-benchmark-dataset-learning-intervene-online/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan…
24. Răzvan Rughinis , , Cosima Rughinis , , and Emanuela Bran. 2024. Generative ai and social engines of hate. In Regulating Hate Speech Created by Generative AI, pages 1–18. Auerbach Publications. [find](https://scholar.google.com/scholar?q=Ra%CC%86zvan+Rughinis+%2C+%2C+Cosima+Rughinis+%2C+%2C+and+Emanuela+Bran%2E+2024%2E+Generative+ai+and+social+engines+of+hate%2E+In+Regulating+Hate+Speech+Created+by+Generative+AI%2C+pages+1%E2%80%9318%2E+Auerbach+Publications%2E)
25. Punyajoy Saha, Aalok Agrawal, Abhik Jana, Chris Biemann, and Animesh Mukherjee. 2024. On zero-shot counterspeech generation by llms. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 12443–12454. [find](https://scholar.google.com/scholar?q=Punyajoy+Saha%2C+Aalok+Agrawal%2C+Abhik+Jana%2C+Chris+Biemann%2C+and+Animesh+Mukherjee%2E+2024%2E+On+zero%2Dshot+counterspeech+generation+by+llms%2E+In+Proceedings+of+the+2024+Joint+International+Conference+on+Comput)
26. Punyajoy Saha, Kanishk Singh, Adarsh Kumar, Binny Mathew, and Animesh Mukherjee. 2022. Coun-tergedi: A controllable approach to generate polite, detoxified and emotional counterspeech. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 5157–… [find](https://scholar.google.com/scholar?q=Punyajoy+Saha%2C+Kanishk+Singh%2C+Adarsh+Kumar%2C+Binny+Mathew%2C+and+Animesh+Mukherjee%2E+2022%2E+Coun%2Dtergedi%3A+A+controllable+approach+to+generate+polite%2C+detoxified+and+emotional+counterspeech%2E+In+Proceedings+)
27. Pranab Kumar Sen. 1968. Estimates of the regression coefficient based on kendall’s tau. Journal of the American statistical association, 63(324):1379– 1389. [find](https://scholar.google.com/scholar?q=Pranab+Kumar+Sen%2E+1968%2E+Estimates+of+the+regression+coefficient+based+on+kendall%E2%80%99s+tau%2E+Journal+of+the+American+statistical+association%2C+63%28324%29%3A1379%E2%80%93+1389%2E)
28. Saha Sougata and Srihari Rohini. 2023. Consolidating strategies for countering hate speech using persuasive dialogues. In Proceedings of the 20th International Conference on Natural Language Processing (ICON), pages 378–392. [find](https://scholar.google.com/scholar?q=Saha+Sougata+and+Srihari+Rohini%2E+2023%2E+Consolidating+strategies+for+countering+hate+speech+using+persuasive+dialogues%2E+In+Proceedings+of+the+20th+International+Conference+on+Natural+Language+Processin)
29. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, and 1 others. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. against online hate speech.… [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Thomas+Mesnard%2C+Cassidy+Hardin%2C+Robert+Dadashi%2C+Surya+Bhupatiraju%2C+Shreya+Pathak%2C+Laurent+Sifre%2C+Morgane+Rivi%C3%A8re%2C+Mihir+Sanjay+Kale%2C+Juliette+Love%2C+and+1+others%2E+2024%2E+Gemma%3A+Open+models+b)

*Updated 2 November 2026*