--- title: "Generating Counter Narratives against Online Hate Speech: Data and Strategies (2020)" description: "To combine their strengths, they propose an author-reviewer pipeline. 3 references, cited by 69 papers here." authors: ["Serra Sinem Tekiroğlu", "Yi-Ling Chung", "Marco Guerini"] year: 2020 venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics" source: "https://aclanthology.org/2020.acl-main.110/" doi: "10.18653/v1/2020.acl-main.110" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/tekiroglu-2020-generating-counter-narratives-against-online/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tekiroglu-2020-generating-counter-narratives-against-online/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / Generating Counter Narratives against Online Hate Speech: Data and Strategies # Generating Counter Narratives against Online Hate Speech: Data and Strategies **Authors:** [Serra Sinem Tekiroğlu](https://md.research.elamur.ai/free/en/authors/serra-sinem-tekiroglu/), [Yi-Ling Chung](https://md.research.elamur.ai/free/en/authors/yi-ling-chung/) and [Marco Guerini](https://md.research.elamur.ai/free/en/authors/marco-guerini/) · **Year:** 2020 · **Venue:** [Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2020](https://md.research.elamur.ai/free/en/venues/acl/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.110/) · [PDF](https://aclanthology.org/2020.acl-main.110.pdf) · [DOI 10.18653/v1/2020.acl-main.110](https://doi.org/10.18653/v1/2020.acl-main.110) · [OpenAlex](https://openalex.org/W3016257480) **Citations:** 84 citations (OpenAlex, 2026-10-14) · [69](https://md.research.elamur.ai/free/en/papers/tekiroglu-2020-generating-counter-narratives-against-online/cited-by/) in our library, of the 141,840 papers whose bibliographies we read ## Summary Counter-narratives (CNs) are calm, fact-based replies to hate speech (HS). They are an alternative to deleting content, but expert-written CNs are slow to produce and too few to train generation models. The authors first compare three existing ways of collecting HS-CN pairs. Crawling from social media (CRAWL) yields many hostile replies. Crowdsourcing (CROWD) gives large volume but shallow, repetitive answers. Nichesourcing with NGO experts (NICHE) is the richest but small and hard to organise. To combine their strengths, they propose an author-reviewer pipeline. A fine-tuned GPT-2 acts as the author and drafts candidate pairs. Crowd annotators or a classifier act as the reviewer and pre-filter them. NGO operators then accept, edit or discard what remains. In a study with five operators, machine-generated suggestions cut expert time per usable pair from 480 to well under 80 seconds. Stricter filtering saves more expert time but makes the CNs more generic and less novel. This gives a scalable recipe for building high-quality training data where experts are scarce. ## Key points - **Diversity of existing datasets:** On 100 sampled pairs per dataset, 50% of CRAWL CNs were hostile. CROWD was 76% pure denouncing. NICHE had 81% richer "other" arguments, such as facts. NICHE also had the lowest Repetition Rate (2.72), compared with 3.16 for CRAWL and 4.83 for CROWD. - **Machine reviewer choice:** ALBERT was selected over BERT, reaching F1 0.73 versus 0.67, with higher precision and recall as well. - **Expert time per pair:** Average NGO operator time per obtained pair was: - 480 s with no suggestions; - 76 s for unfiltered GPT-2 output; - 72 s with crowd filter ≥1; - 68 s with the ALBERT filter; - 49 s with crowd filter ≥2. - **Acceptance rates:** Experts accepted or edited 45% of unfiltered candidates, rising to 72% under the strictest crowd filter. That filter, however, costs about 700 s of crowd time per surviving pair. - **Editing effort:** HTER of operator edits stayed between 0.1 and 0.2, well below the 0.4 acceptability threshold. This suggests operators only kept candidates that were easy to fix. ## Interesting ideas - **Two-sided data quality.** Quality is split into conformity to guidelines and diversity, both lexical and semantic, with semantic diversity measured through argument-type annotation. This checklist could be reused to audit any crowdsourced or generated corpus beyond simple size counts. - **Repetition Rate for comparing corpora.** Repetitiveness is measured with Repetition Rate over sliding 1000-word windows, after deduplicating and shuffling. Because the windows are fixed in size, corpora of different sizes can be compared fairly, which plain distinct-n-gram counts do not allow. - **Author-reviewer pipeline for silver data.** A generator produces candidates, cheap reviewers (crowd or classifier) filter them, and experts only post-edit. This pattern could speed up dataset creation for low-resource languages or other specialist domains. - **Filtering pushes toward the generic.** Strict agreement-based filtering favours bland, generic outputs that reviewers easily agree on, which lowers novelty. Any pipeline that uses inter-annotator agreement as a selection criterion should track diversity alongside acceptance rate. - **Measuring cost where it matters.** The evaluation uses expert time per usable item as the main metric, adds crowd cost and post-editing effort (HTER), and randomises condition order. This is a practical template for judging human-in-the-loop tools by real effort saved. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Fondazione Bruno Kessler](https://md.research.elamur.ai/free/en/labs/fondazione-bruno-kessler/) — Research institute, Trento, Italy · 25 works · 43,498 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 744 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 221 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 221 works 🧭 Topics (3): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 608 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 209 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 233 works ## Cite Cite as: Serra Sinem Tekiroğlu, Yi-Ling Chung and Marco Guerini. 2020. Generating Counter Narratives against Online Hate Speech: Data and Strategies. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{tekiroglu2020generating, title = {{Generating Counter Narratives against Online Hate Speech: Data and Strategies}}, author = {Serra Sinem Tekiroğlu, Yi-Ling Chung and Marco Guerini}, booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics}, year = {2020}, doi = {10.18653/v1/2020.acl-main.110}, url = {https://doi.org/10.18653/v1/2020.acl-main.110} } ``` - [← previous Fill in the BLANC: Human-free quality estimation of document summaries](https://md.research.elamur.ai/free/en/papers/vasilyev-2020-fill-blanc-human-free-quality/) - [next → Unsupervised Quality Estimation for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/fomicheva-2020-unsupervised-quality-estimation-neural-machine/) - [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) ## References 3 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Susan Benesch. 2014. Countering dangerous speech: New ideas for genocide prevention. Washington, DC: United States Holocaust Memorial Museum. [find](https://scholar.google.com/scholar?q=Susan+Benesch%2E+2014%2E+Countering+dangerous+speech%3A+New+ideas+for+genocide+prevention%2E+Washington%2C+DC%3A+United+States+Holocaust+Memorial+Museum%2E) 2. Susan Benesch, Derek Ruths, Kelly P Dillon, Haji Mohammad Saleem, and Lucas Wright. 2016. Counterspeech on twitter: A field study. Dangerous Speech Project. Available at: [https://dangerousspeech.org/counterspeech-on-twitter-a-field-study/](https://dangerousspeech.org/counterspeech-on-twitter-a-field-study/). [↗](https://dangerousspeech.org/counterspeech-on-twitter-a-field-study/) 3. Nicola Bertoldi, Mauro Cettolo, and Marcello Federico. 2013. Cache-based online adaptation for machine translation enhanced computer assisted translation. In MT-Summit, pages 35–42. \#\# Conclusions [find](https://scholar.google.com/scholar?q=Nicola+Bertoldi%2C+Mauro+Cettolo%2C+and+Marcello+Federico%2E+2013%2E+Cache%2Dbased+online+adaptation+for+machine+translation+enhanced+computer+assisted+translation%2E+In+MT%2DSummit%2C+pages+35%E2%80%9342%2E+%5C%23%5C%23+Conclusions) *Updated 24 October 2026*