---
title: "Hateful People or Hateful Bots? Detection and — Albadi 2019"
description: "The paper asks how much of the religious hate speech on Arabic Twitter comes from automated accounts rather than people."
authors: ["Nuha Albadi", "Maram Kurdi", "Shivakant Mishra"]
year: 2019
venue: "arXiv:1908.00153 · Proc. ACM Hum.-Comput. Interact. 3, CSCW: Article 61 (2019)"
source: "https://arxiv.org/abs/1908.00153"
doi: "10.1145/3359163"
arxiv: "1908.00153"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/albadi-2019-hateful-people-hateful-bots-detection/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/albadi-2019-hateful-people-hateful-bots-detection/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / Hateful People or Hateful Bots? Detection and Characterization of Bots Spreading Religious Hatred in Arabic Social Media

# Hateful People or Hateful Bots? Detection and Characterization of Bots Spreading Religious Hatred in Arabic Social Media

**Authors:** Nuha Albadi, Maram Kurdi, Shivakant Mishra · **Institutions:** [University of Colorado Boulder](https://md.research.elamur.ai/free/en/universities/university-of-colorado-boulder/) · [Taibah University](https://md.research.elamur.ai/free/en/universities/taibah-university/) · [Taif University](https://md.research.elamur.ai/free/en/universities/taif-university/) · **Year:** 2019 · **Venue:** arXiv:1908.00153 · Proc. ACM Hum.-Comput. Interact. 3, CSCW: Article 61 (2019) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://doi.org/10.1145/3359163) · [PDF](https://arxiv.org/pdf/1908.00153) · [DOI 10.1145/3359163](https://doi.org/10.1145/3359163) · [arXiv 1908.00153](https://arxiv.org/abs/1908.00153) · [OpenAlex](https://openalex.org/W3125571733)

**Citations:** 10 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/albadi-2019-hateful-people-hateful-bots-detection/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper asks how much of the religious hate speech on Arabic Twitter comes from automated accounts rather than people. It also tests whether existing bot detectors, built mostly on English data, still work for Arabic. The authors took a sample of 450 accounts from their earlier Arabic hate-speech dataset. They hand-coded each account with a bot-likelihood score and compared these scores against Botometer, which turned out only moderately reliable on Arabic accounts. They then trained their own regression model on 86,346 tweets from those accounts, using linguistic, content, behavioral and topic features. The model beat Botometer by 31 points in Spearman's rho. Even a version restricted to simple content and linguistic cues gained 20 points over it. Bots produced roughly 11% of the hateful tweets. The work matters because social-media studies often assume every post has a human author. It shows that language-specific signals are needed when detection moves beyond English, and the labeled accounts are released for others.

## Key points
- 450 manually labeled accounts that spread religious hate, with 86,346 tweets, published at github.com/nuhaalbadi/ArabicBots.
- Bots account for about 11% of hateful tweets in the authors' Arabic hate-speech dataset.
- Botometer performs only a little above average on Arabic bots. The proposed regression model exceeds it by 31 points in Spearman's rho, and by 20 points using content and linguistic features alone.
- Compared with humans, the bots:
  - converse less with other accounts;
  - post more often through third-party applications;
  - cover a narrower range of topics;
  - use fewer emojis and more numerals.
- Average word length and punctuation counts are also informative. The authors note the model will need regular retraining as bot behavior drifts.

## Interesting ideas
- Bot status is labeled and predicted as a graded likelihood rather than a yes/no class, and evaluation uses rank correlation (Spearman's rho), which rewards getting the ordering of uncertain cases right. — For Mova (learn): parser and frame confidence scores could be checked by rank correlation against graded human or LLM-teacher judgments, not only by accuracy on hard labels.
- Running a strong off-the-shelf English tool on a new language exposes a hidden transfer gap that headline accuracy would never show. — For Mova (learn): every component should be scored on out-of-domain text (tales vs. textbooks vs. web) as a standing control, so domain-shift drops become visible.
- Cheap surface statistics, such as emoji and numeral rates, punctuation density and mean word length, carried much of the discriminative signal. — For Mova (tokens): the tokenizer could emit these counts as explainable genre/register signals that downstream stages consult before parsing noisy input.
- Features were grouped into families (linguistic, content, behavioral, topic) and compared as sets, showing which kind of evidence drives the result. — For Mova (learn): ablating whole feature families or rule groups in the parser and repair rules gives an interpretable account of what each module contributes.
- Narrow topical range was itself a marker of automated authorship. — For Mova (world): measuring how varied the entities and events in a text's world are could flag templated or generated input that deserves lower trust.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{albadi2019hateful,
  title = {{Hateful People or Hateful Bots? Detection and Characterization of Bots Spreading Religious Hatred in Arabic Social Media}},
  author = {Albadi, Nuha and Kurdi, Maram and Mishra, Shivakant},
  year = {2019},
  doi = {10.1145/3359163},
  eprint = {1908.00153},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3359163}
}
```

- [← previous Evaluation of named entity coreference](https://md.research.elamur.ai/free/en/papers/agarwal-2019-evaluation-named-entity-coreference/)
- [next → Sentiment Analysis for Arabic in Social Media Network: A Systematic Mapping Study](https://md.research.elamur.ai/free/en/papers/abo-2019-sentiment-analysis-arabic-social-media/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/)

## References

60 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Norah Abokhodair, Daisy Yoo, and David W McDonald. 2015. Dissecting a social botnet: Growth, content and influence in Twitter. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing. ACM, 839–851. https://doi.org/10.1145/2675133.2675208 [↗](https://doi.org/10.1145/2675133.2675208)
2. Aseel Addawood, Adam Badawy, Kristina Lerman, and Emilio Ferrara. 2019. Linguistic Cues to Deception: Identifying Political Trolls on Social Media. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 13. AAAI, 15–25. [https://www.aaai.org/ojs/index.php/ICWSM/article/view/3205](https://www.aaai.org/ojs/index.php/ICWSM/article/view/3205) [↗](https://www.aaai.org/ojs/index.php/ICWSM/article/view/3205)
3. Nuha Albadi, Maram Kurdi, and Shivakant Mishra. 2018. Are they Our Brothers? Analysis and Detection of Religious Hate Speech in the Arabic Twittersphere. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). IEEE, 69–76. https://doi.org/10.1109/ASONAM.2018.8508247 [↗](https://doi.org/10.1109/asonam.2018.8508247)
4. Nuha Albadi, Maram Kurdi, and Shivakant Mishra. 2019. Investigating the effect of combining GRU neural networks with handcrafted features for religious hatred detection on Arabic Twitter space. Social Network Analysis and Mining 9, 1 (05 Aug 2019), 41. https://doi.org/10.1007/s13278-019-0587-5 [↗](https://doi.org/10.1007/s13278-019-0587-5)
5. Pinkesh Badjatiya, Shashank Gupta, Manish Gupta, and Vasudeva Varma. 2017. Deep learning for hate speech detection in tweets. In Proceedings of the 26th International Conference on World Wide Web Companion. International World Wide Web Conferences Steering Committee, 759–760. https://doi.org/10.1145/3041021.3054223 [↗](https://doi.org/10.1145/3041021.3054223)
6. Matthew C Benigni, Kenneth Joseph, and Kathleen M Carley. 2017. Online extremism and the communities that sustain it: Detecting the ISIS supporting community on Twitter. PloS one 12, 12 (2017), e0181405. https://doi.org/10.1371/jour nal.pone.0181405 [↗](https://doi.org/10.1371/jour)
7. Jonathon M Berger and Jonathon Morgan. 2015. The ISIS Twitter Census: Defining and describing the population of ISIS supporters on Twitter. The Brookings Project on US Relations with the Islamic World 3, 20 (2015), 4–1. [https://www.brookings.edu/wp-content/uploads/2016/06/isis_twitter_census_berger_morgan.pdf](https://www.brookings.edu/wp-content/uploads/2016/06/isis_twitter_census_berger_morgan.pdf) [↗](https://www.brookings.edu/wp-content/uploads/2016/06/isis_twitter_census_berger_morgan.pdf)
8. David M Beskow and Kathleen M Carley. 2018. Bot conversations are different: leveraging network metrics for bot detection in twitter. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). IEEE, 825–832. https://doi.org/10.1109/asonam.2018.8508322 [↗](https://doi.org/10.1109/asonam.2018.8508322)
9. Alessandro Bessi and Emilio Ferrara. 2016. Social bots distort the 2016 US Presidential election online discussion. First Monday 21, 11 (2016). https://doi.org/10.5210/fm.v21i11.7090 [↗](https://doi.org/10.5210/fm.v21i11.7090)
10. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. " O’Reilly Media, Inc.". https://doi.org/10.1007/s10579-010-9124-x [↗](https://doi.org/10.1007/s10579-010-9124-x)
11. Javier Borge-Holthoefer, Walid Magdy, Kareem Darwish, and Ingmar Weber. 2015. Content and network dynamics behind Egyptian political polarization on Twitter. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing. ACM, 700–711. https://doi.org/10.1145/2675133.2675163 [↗](https://doi.org/10.1145/2675133.2675163)
12. Stevie Chancellor, Andrea Hu, and Munmun De Choudhury. 2018. Norms matter: contrasting social support around behavior change in online weight loss communities. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 666. https://doi.org/10.1145/3173574.3174240 [↗](https://doi.org/10.1145/3173574.3174240)
13. Eshwar Chandrasekharan, Umashanthi Pavalanathan, Anirudh Srinivasan, Adam Glynn, Jacob Eisenstein, and Eric Gilbert. 2017. You can’t stay here: The efficacy of reddit’s 2015 ban examined through hate speech. Proceedings of the ACM on Human-Computer Interaction 1, CSCW (2017), 31. https://doi.org/10.1145/3134666 [↗](https://doi.org/10.1145/3134666)
14. Nikan Chavoshi, Hossein Hamooni, and Abdullah Mueen. 2017. On-demand bot detection and archival system. In Proceedings of the 26th International Conference on World Wide Web Companion. International World Wide Web Conferences Steering Committee, 183–187. https://doi.org/10.1145/3041021.3054733 [↗](https://doi.org/10.1145/3041021.3054733)
15. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin 70, 4 (1968), 213. https://doi.org/10.1037/h0026256 [↗](https://doi.org/10.1037/h0026256)
16. Christian S. Crandall and Charles Stangor. 2008. Conformity and Prejudice. John Wiley & Sons, Ltd, Chapter 18, 293–309. https://doi.org/10.1002/9780470773963.ch18 [↗](https://doi.org/10.1002/9780470773963.ch18)
17. Kareem Darwish, Walid Magdy, and Ahmed Mourad. 2012. Language processing for arabic microblog retrieval. In Proceedings of the 21st ACM international conference on Information and knowledge management. ACM, 2427–2430. https://doi.org/10.1145/2396761.2398658 [↗](https://doi.org/10.1145/2396761.2398658)
18. Clayton Allen Davis, Onur Varol, Emilio Ferrara, Alessandro Flammini, and Filippo Menczer. 2016. Botornot: A system to evaluate social bots. In Proceedings of the 25th International Conference Companion on World Wide Web. International World Wide Web Conferences Steering Committee, 273–274. https://doi.org/10.1145/2872518.2889302 [↗](https://doi.org/10.1145/2872518.2889302)
19. Munmun De Choudhury, Sanket S Sharma, Tomaz Logar, Wouter Eekhout, and René Clausen Nielsen. 2017. Gender and cross-cultural differences in social media disclosures of mental illness. In Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing. ACM, 353–369. https://doi.org/10.1145/2998181.2998220 [↗](https://doi.org/10.1145/2998181.2998220)
20. Mona Diab, Kadri Hacioglu, and Daniel Jurafsky. 2004. Automatic tagging of Arabic text: From raw text to base phrase chunks. In Proceedings of HLT-NAACL 2004: Short papers. Association for Computational Linguistics, 149–152. https://doi.org/10.3115/1613984.1614022 [↗](https://doi.org/10.3115/1613984.1614022)
21. Nemanja Djuric, Jing Zhou, Robin Morris, Mihajlo Grbovic, Vladan Radosavljevic, and Narayan Bhamidipati. 2015. Hate speech detection with comment embeddings. In Proceedings of the 24th international conference on world wide web. ACM, 29–30. https://doi.org/10.1145/2740908.2742760 [↗](https://doi.org/10.1145/2740908.2742760)
22. Nour El-Mawass and Saad Alaboodi. 2016. Detecting Arabic spammers and content polluters on Twitter. In 2016 Sixth International Conference on Digital Information Processing and Communications (ICDIPC). IEEE, 53–58. https://doi.org/10.1109/icdipc.2016.7470791 [↗](https://doi.org/10.1109/icdipc.2016.7470791)
23. Sindhu Kiranmai Ernala, Asra F Rizvi, Michael L Birnbaum, John M Kane, and Munmun De Choudhury. 2017. Linguistic markers indicating therapeutic outcomes of social media disclosures of schizophrenia. Proceedings of the ACM on Human-Computer Interaction 1, CSCW (2017), 43. https://doi.org/10.1145/3134678 [↗](https://doi.org/10.1145/3134678)
24. Ali Farghaly and Khaled Shaalan. 2009. Arabic natural language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing (TALIP) 8, 4 (2009), 14. https://doi.org/10.1145/1644879.1644881 [↗](https://doi.org/10.1145/1644879.1644881)
25. Emilio Ferrara, Onur Varol, Clayton Davis, Filippo Menczer, and Alessandro Flammini. 2016. The rise of social bots. Commun. ACM 59, 7 (2016), 96–104. https://doi.org/10.1145/2818717 [↗](https://doi.org/10.1145/2818717)
26. Geoff Golberg. 2018. When it comes to dealing with fake/bot accounts, Twitter is (still) failing. Retrieved March 04, 2019 from [https://medium.com/@geoffgolberg/when-it-comes-to-dealing-with-fake-bot-accounts-twitter-is-still-fai](https://medium.com/@geoffgolberg/when-it-comes-to-dealing-with-fake-bot-accounts-twitter-is-still-fai) ling-a79d9ece5b5d [↗](https://medium.com/@geoffgolberg/when-it-comes-to-dealing-with-fake-bot-accounts-twitter-is-still-fai)
27. Andrew Hall, Loren Terveen, and Aaron Halfaker. 2018. Bot Detection in Wikidata Using Behavioral and Other Informal Cues. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 64. https://doi.org/10.1145/3274333 [↗](https://doi.org/10.1145/3274333)
28. Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika 30, 1/2 (1938), 81–93. https://doi.org/10.2307/2332226 [↗](https://doi.org/10.2307/2332226)
29. John Vines Kiel Long, Selina Sutton, Phillip Brooker, Tom Feltwell, Ben Kirman, Julie Barnett, and Shaun Lawson. 2017. Could You Define That in Bot Terms?: Requesting, Creating and Using Bots on Reddit. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 3488–3500. https://doi.org/10.1145/3025453.3025830 [↗](https://doi.org/10.1145/3025453.3025830)
30. RoBhat Labs. 2017. botcheck.me: detect & track Twitter bots. Retrieved March 05, 2019 from [https://botcheck.me](https://botcheck.me) [↗](https://botcheck.me)
31. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159–174. https://doi.org/10.2307/2529310 [↗](https://doi.org/10.2307/2529310)
32. Kyumin Lee, Brian David Eoff, and James Caverlee. 2011. Seven months with the devils: A long-term study of content polluters on twitter. In Fifth International AAAI Conference on Weblogs and Social Media. AAAI, 185–192. [https://www.aaai.org/ocs/index.php/ICWSM/ICWSM11/paper/viewPaper/2780](https://www.aaai.org/ocs/index.php/ICWSM/ICWSM11/paper/viewPaper/2780) [↗](https://www.aaai.org/ocs/index.php/ICWSM/ICWSM11/paper/viewPaper/2780)
33. Neil Liberman. 2017. Decision trees and random forests. Retrieved March 31, 2019 from [https://towardsdatascience.com/decision-trees-and-random-forests-df0c3123f991](https://towardsdatascience.com/decision-trees-and-random-forests-df0c3123f991) [↗](https://towardsdatascience.com/decision-trees-and-random-forests-df0c3123f991)
34. Steven Loria. 2018. textblob Documentation. Retrieved March 16, 2019 from [https://media.readthedocs.org/pdf/textbl](https://media.readthedocs.org/pdf/textbl) ob/latest/textblob.pdf [↗](https://media.readthedocs.org/pdf/textbl)
35. Walid Magdy, Kareem Darwish, Norah Abokhodair, Afshin Rahimi, and Timothy Baldwin. 2016. # isisisnotislam or# deportallmuslims?: Predicting unspoken views. In Proceedings of the 8th ACM Conference on Web Science. ACM, 95–106. https://doi.org/10.1145/2908131.2908150 [↗](https://doi.org/10.1145/2908131.2908150)
36. Mainack Mondal, Leandro Araújo Silva, and Fabrício Benevenuto. 2017. A measurement study of hate speech in social media. In Proceedings of the 28th ACM Conference on Hypertext and Social Media. ACM, 85–94. https://doi.org/10.1145/3078714.3078723 [↗](https://doi.org/10.1145/3078714.3078723)
37. Fred Morstatter, Liang Wu, Tahora H Nazer, Kathleen M Carley, and Huan Liu. 2016. A new approach to bot detection: striking the balance between precision and recall. In 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). IEEE, 533–540. https://doi.org/10.1109/asonam.2016.7752287 [↗](https://doi.org/10.1109/asonam.2016.7752287)
38. Elizabeth Levy Paluck and Donald P Green. 2009. Prejudice reduction: What works? A review and assessment of research and practice. Annual review of psychology 60 (2009), 339–367. https://doi.org/10.1146/annurev.psych.60.110707.163607 [↗](https://doi.org/10.1146/annurev.psych.60.110707.163607)
39. Umashanthi Pavalanathan, Xiaochuang Han, and Jacob Eisenstein. 2018. Mind Your POV: Convergence of Articles and Editors Towards Wikipedia’s Neutrality Norm. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 137. https://doi.org/10.1145/3274406 [↗](https://doi.org/10.1145/3274406)
40. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825–2830. [http://www.jmlr.org/papers/v12/pedr](http://www.jmlr.org/papers/v12/pedr) egosa11a.html [↗](http://www.jmlr.org/papers/v12/pedr)
41. Pew Research Center, Washington, D.C. 2018. 5 things to know about bots on Twitter. Retrieved March 07, 2019 from [http://www.pewresearch.org/fact-tank/2018/04/09/5-things-to-know-about-bots-on-twitter/](http://www.pewresearch.org/fact-tank/2018/04/09/5-things-to-know-about-bots-on-twitter/) [↗](http://www.pewresearch.org/fact-tank/2018/04/09/5-things-to-know-about-bots-on-twitter/)
42. Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher D. Manning. 2018. Universal Dependency Parsing from Scratch. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics, Brussels, Belgium, 160–170. [https://nlp.stanford.edu/pubs/qi2018universal.pdf](https://nlp.stanford.edu/pubs/qi2018universal.pdf) [↗](https://nlp.stanford.edu/pubs/qi2018universal.pdf) [our card](https://md.research.elamur.ai/free/en/papers/ud-2018-zeman-conll-2018-shared-task-multilingual/)
43. Jacob Ratkiewicz, Michael Conover, Mark Meiss, Bruno Gonçalves, Snehal Patil, Alessandro Flammini, and Filippo Menczer. 2011. Truthy: mapping the spread of astroturf in microblog streams. In Proceedings of the 20th international conference companion on world wide web. ACM, 249–252. https://doi.org/10.1145/1963192.1963301 [↗](https://doi.org/10.1145/1963192.1963301)
44. Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks. ELRA, Valletta, Malta, 45–50. [http://is.muni.cz/publi](http://is.muni.cz/publi) cation/884893/en. [↗](http://is.muni.cz/publi)
45. Eugenia Ha Rim Rho, Gloria Mark, and Melissa Mazmanian. 2018. Fostering Civil Discourse Online: Linguistic Behavior in Comments of# MeToo Articles across Political Perspectives. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 147. https://doi.org/10.1145/3274416 [↗](https://doi.org/10.1145/3274416)
46. Ando Saabas. 2014. Selecting good features - part III: random forests. Retrieved March 20, 2019 from [http://blog.datad](http://blog.datad) ive.net/selecting-good-features-part-iii-random-forests/ [↗](http://blog.datad)
47. Ando Saabas. 2015. Random forest interpretation with scikit-learn. Retrieved March 14, 2019 from [http://blog.datadiv](http://blog.datadiv) e.net/random-forest-interpretation-with-scikit-learn/ [↗](http://blog.datadiv)
48. Joseph Seering, Juan Pablo Flores, Saiph Savage, and Jessica Hammer. 2018. The Social Roles of Bots: Evaluating Impact of Bots on Discussions in Online Communities. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 157. https://doi.org/10.1145/3274426 [↗](https://doi.org/10.1145/3274426)
49. Leandro Araújo Silva, Mainack Mondal, Denzil Correa, Fabrício Benevenuto, and Ingmar Weber. 2016. Analyzing the Targets of Hate in Online Social Media.. In Proceedings of the 11th International AAAI Conference on Web and Social Media (ICWSM ’16). AAAI, 687–690. [https://www.aaai.org/ocs/index.php/ICWSM/ICWSM16/paper/viewPaper/13147](https://www.aaai.org/ocs/index.php/ICWSM/ICWSM16/paper/viewPaper/13147) [↗](https://www.aaai.org/ocs/index.php/ICWSM/ICWSM16/paper/viewPaper/13147)
50. Devin Soni and Vivek K Singh. 2018. See No Evil, Hear No Evil: Audio-Visual-Textual Cyberbullying Detection. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 164. https://doi.org/10.1145/3274433 [↗](https://doi.org/10.1145/3274433)
51. Charles Spearman. 1904. The proof and measurement of association between two things. American journal of Psychology 15, 1 (1904), 72–101. https://doi.org/10.1037/11491-005 [↗](https://doi.org/10.1037/11491-005)
52. Charles Stangor, Gretchen B Sechrist, and John T Jost. 2001. Changing racial beliefs by providing consensus information. Personality and Social Psychology Bulletin 27, 4 (2001), 486–496. https://doi.org/10.1177/0146167201274009 [↗](https://doi.org/10.1177/0146167201274009)
53. Stefan Stieglitz, Florian Brachten, Davina Berthelé, Mira Schlaus, Chrissoula Venetopoulou, and Daniel Veutgen. 2017. Do social bots (still) act different to humans?–comparing metrics of social bots with those of humans. In International Conference on Social Computing and Social Media. Springer, 379–395. https://doi.org/10.1007/978-3-319-58559-8_30 [↗](https://doi.org/10.1007/978-3-319-58559-8_30)
54. Gianluca Stringhini, Christopher Kruegel, and Giovanni Vigna. 2010. Detecting spammers on social networks. In Proceedings of the 26th annual computer security applications conference. ACM, 1–9. https://doi.org/10.1145/1920261.1920263 [↗](https://doi.org/10.1145/1920261.1920263)
55. Twitter. 2018. How Twitter is fighting spam and malicious automation. Retrieved March 03, 2019 from [https://blog.t](https://blog.t) witter.com/official/en_us/topics/company/2018/how-twitter-is-fighting-spam-and-malicious-automation.html [↗](https://blog.t)
56. Twitter. 2018. Update on TwitterâĂŹs review of the 2016 US election. Retrieved March 03, 2019 from [https://blog.twitter.com/official/en_us/topics/company/2018/2016-election-update.html](https://blog.twitter.com/official/en_us/topics/company/2018/2016-election-update.html) [↗](https://blog.twitter.com/official/en_us/topics/company/2018/2016-election-update.html)
57. Twitter. 2019. Number of monthly active Twitter users worldwide from 1st quarter 2010 to 4th quarter 2018 (in millions). Retrieved March 03, 2019 from [https://www.statista.com/statistics/282087/number-of-monthly-active-twitter-users/](https://www.statista.com/statistics/282087/number-of-monthly-active-twitter-users/) [↗](https://www.statista.com/statistics/282087/number-of-monthly-active-twitter-users/)
58. Onur Varol, Emilio Ferrara, Clayton A Davis, Filippo Menczer, and Alessandro Flammini. 2017. Online human-bot interactions: Detection, estimation, and characterization. In Eleventh international AAAI conference on web and social media. AAAI, 280–289. [https://aaai.org/ocs/index.php/ICWSM/ICWSM17/paper/view/15587](https://aaai.org/ocs/index.php/ICWSM/ICWSM17/paper/view/15587) [↗](https://aaai.org/ocs/index.php/ICWSM/ICWSM17/paper/view/15587)
59. Gang Wang, Manish Mohanlal, Christo Wilson, Xiao Wang, Miriam Metzger, Haitao Zheng, and Ben Y Zhao. 2012. Social Turing Tests: Crowdsourcing Sybil Detection. In Proceedings of the Network and Distributed System Security Symposium (NDSS). The Internet Society, 1–16. [https://www.ndss-symposium.org/ndss2013/ndss-2013-programme/s](https://www.ndss-symposium.org/ndss2013/ndss-2013-programme/s) ocial-turing-tests-crowdsourcing-sybil-detection/ [↗](https://www.ndss-symposium.org/ndss2013/ndss-2013-programme/s)
60. Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL-HLT ’16). Association for Computational Linguistics, 88–93. https://doi.org/10.18653/v1/n16-2013 [↗](https://doi.org/10.18653/v1/n16-2013) [our card](https://md.research.elamur.ai/free/en/papers/waseem-2016-hateful-symbols-or-hateful-people/)