---
title: "Overview of the Evalita 2018 Task on — Fersini 2018"
description: "This overview describes AMI, a shared task at Evalita 2018 on detecting misogyny in Italian and English tweets."
authors: ["Elisabetta Fersini", "Debora Nozza", "Paolo Rosso"]
year: 2018
venue: "Proceedings of the Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2018)"
source: "https://aclanthology.org/2018.evalita-1.10/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/fersini-2018-overview-evalita-2018-task-automatic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/fersini-2018-overview-evalita-2018-task-automatic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/) / Overview of the Evalita 2018 Task on Automatic Misogyny Identification (AMI)

# Overview of the Evalita 2018 Task on Automatic Misogyny Identification (AMI)

**Authors:** [Elisabetta Fersini](https://md.research.elamur.ai/free/en/authors/elisabetta-fersini/), [Debora Nozza](https://md.research.elamur.ai/free/en/authors/debora-nozza/), [Paolo Rosso](https://md.research.elamur.ai/free/en/authors/paolo-rosso/) · **Year:** 2018 · **Venue:** Proceedings of the Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2018) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2018.evalita-1.10/) · [PDF](https://aclanthology.org/2018.evalita-1.10.pdf) · [OpenAlex](https://openalex.org/W2906979176)

**Citations:** 213 citations (openalex-snapshot · 2026-10-14) · [101](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
This overview describes AMI, a shared task at Evalita 2018 on detecting misogyny in Italian and English tweets. It extends an earlier IberEval 2018 edition. Subtask A is binary: is a tweet misogynous or not. Subtask B asks systems to assign each misogynous tweet one of five behaviour types (Stereotype & Objectification, Dominance, Derailing, Sexual Harassment & Threats of Violence, Discredit) and to say whether it targets an individual (active) or a group (passive).

The organisers built the data in three ways: keyword streaming, watching accounts of likely victims, and pulling the timelines of self-declared misogynists. Six experts labelled the tweets on CrowdFlower. Subtask A was scored with accuracy. Subtask B used the mean of two macro-F1 scores, one for behaviour and one for target. The reference system, AMI-BASELINE, was an SVM on unigram features.

Six teams took part for Italian and ten for English. Binary detection worked reasonably well, with best accuracy 0.844 for Italian and 0.704 for English. Fine-grained classification stayed hard: most runs fell below the baseline. The resource and its results give a reference point for gender-targeted hate speech research.

## Key points
- **Data size and agreement.** Each language started from about 10,000 candidate tweets and ended with 4,000 training and 1,000 test tweets.
  - Agreement for English (misogynous / category / target) was 0.81 / 0.45 / 0.49.
  - Agreement for Italian was 0.96 / 0.68 / 0.76.
- **Label balance.** The binary label is roughly balanced. Category and target labels are skewed:
  - Italian is dominated by Stereotype & Objectification.
  - English is dominated by Discredit.
  - Active targets dominate, except in the English training set, which is nearly balanced on target.
- **Participation.** Subtask A received 13 Italian and 26 English runs; Subtask B received 11 Italian and 23 English runs. Teams came from 10 countries.
- **Top systems.** All top results were constrained runs.
  - Subtask A, Italian: bakarov (TF-IDF + SVD + Boosting), accuracy 0.844.
  - Subtask A, English: hateminers (sentence embeddings, TF-IDF and averaged word embeddings, fed to Logistic Regression), accuracy 0.704.
  - Subtask B: best Average Macro F1 was about 0.50 for Italian and 0.406 for English (himani).
- **External resources used by teams:**
  - lexicons: HurtLex, "Le parole per ferire" (De Mauro), the noswearing dictionary, a sexist slur list, SentiWordNet;
  - embeddings: GloVe for English and TWITA-based embeddings for Italian.

## Interesting ideas
- **Collect data from three angles.** Combining keyword search with monitoring of probable victims and the histories of known offenders captures abusive text that keyword queries alone would miss. This seed-account approach could help build abuse corpora in other languages, such as Ukrainian.
- **Layered annotation.** Labels are nested: a binary decision first, then behaviour type and target only for positive tweets. This lets one dataset support both an easy task and a hard one, and makes clear where systems break down.
- **Ranking by averaged macro-F1.** Scoring each field separately with macro-F1 and averaging the two keeps small classes like Dominance and Derailing from being drowned out. It suits any multi-field task with skewed labels.
- **A deliberately simple baseline.** The unigram SVM showed that most sophisticated Subtask B systems failed to beat it. Including such a baseline is a cheap way to expose whether a task is genuinely hard.
- **Separate constrained and unconstrained runs.** Distinguishing runs that use only the provided data and lexicons from those that add external training data makes comparisons fairer. It also isolates how much extra resources actually contribute.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

101 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 94 not yet.

- [ArMIS - The Arabic Misogyny and Sexism Corpus with Annotator Subjective Disagreements](https://md.research.elamur.ai/free/en/papers/almanea-2022-armis-arabic-misogyny-sexism-corpus/)
- [Overview of the EVALITA 2018 Hate Speech Detection Task](https://md.research.elamur.ai/free/en/papers/bosco-2018-overview-evalita-2018-hate-speech/)
- [CONAN - COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speech](https://md.research.elamur.ai/free/en/papers/chung-2019-conan-counter-narratives-through-nichesourcing/)
- [A Corpus of Turkish Offensive Language on Social Media](https://md.research.elamur.ai/free/en/papers/coltekin-2020-corpus-turkish-offensive-language-social/)
- [SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification](https://md.research.elamur.ai/free/en/papers/fersini-2022-semeval-2022-task-5-multimedia/)
- [Toxic, Hateful, Offensive or Abusive? What Are We Really Classifying? An Empirical Analysis of Hate Speech Datasets](https://md.research.elamur.ai/free/en/papers/fortuna-2020-toxic-hateful-offensive-or-abusive/)
- [SemEval-2023 Task 10: Explainable Detection of Online Sexism](https://md.research.elamur.ai/free/en/papers/kirk-2023-semeval-2023-task-10-explainable/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{fersini2018overview,
  title = {{Overview of the Evalita 2018 Task on Automatic Misogyny Identification (AMI)}},
  author = {Fersini, Elisabetta and Nozza, Debora and Rosso, Paolo},
  booktitle = {Proceedings of the Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2018)},
  year = {2018},
  doi = {10.4000/books.aaccademia.4497},
  url = {https://doi.org/10.4000/books.aaccademia.4497}
}
```

- [← previous Overview of the EVALITA 2018 Hate Speech Detection Task](https://md.research.elamur.ai/free/en/papers/bosco-2018-overview-evalita-2018-hate-speech/)
- [next → Overview of the EVALITA 2018 Task on Irony Detection in Italian Tweets (IronITA)](https://md.research.elamur.ai/free/en/papers/cignarella-2018-overview-evalita-2018-task-irony/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/)

## References

32 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Resham Ahluwalia, Himani Soni, Edward Callow, Anderson Nascimento, and Martine De Cock. 2018. Detecting Hate Speech Against Women in English Tweets. In Proceedings of Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. [find](https://scholar.google.com/scholar?q=Resham+Ahluwalia%2C+Himani+Soni%2C+Edward+Callow%2C+Anderson+Nascimento%2C+and+Martine+De+Cock%2E+2018%2E+Detecting+Hate+Speech+Against+Women+in+English+Tweets%2E+In+Proceedings+of+Sixth+Evaluation+Campaign+of+Natu)
2. Final Workshop (EVALITA 2018), Turin, Italy. CEUR.org. [find](https://scholar.google.com/scholar?q=Final+Workshop+%28EVALITA+2018%29%2C+Turin%2C+Italy%2E+CEUR%2Eorg%2E)
3. Maria Anzovino, Elisabetta Fersini, and Paolo Rosso. 2018. Automatic Identification and Classification of Misogynistic Language on Twitter. In International Conference on Applications of Natural Language to Information Systems, pages 57–64. Springer. [find](https://scholar.google.com/scholar?q=Maria+Anzovino%2C+Elisabetta+Fersini%2C+and+Paolo+Rosso%2E+2018%2E+Automatic+Identification+and+Classification+of+Misogynistic+Language+on+Twitter%2E+In+International+Conference+on+Applications+of+Natural+Langu)
4. Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2010. Sentiwordnet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining. In Proceedings of the International Conference on Language Resources and Evaluation. [find](https://scholar.google.com/scholar?q=Stefano+Baccianella%2C+Andrea+Esuli%2C+and+Fabrizio+Sebastiani%2E+2010%2E+Sentiwordnet+3%2E0%3A+An+enhanced+lexical+resource+for+sentiment+analysis+and+opinion+mining%2E+In+Proceedings+of+the+International+Conferen) [our card](https://md.research.elamur.ai/free/en/papers/baccianella-2010-sentiwordnet-3-0-enhanced-lexical/)
5. Amir Bakarov. 2018. Vector Space Models for Automatic Misogyny Identification. In Proceedings of Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. [find](https://scholar.google.com/scholar?q=Amir+Bakarov%2E+2018%2E+Vector+Space+Models+for+Automatic+Misogyny+Identification%2E+In+Proceedings+of+Sixth+Evaluation+Campaign+of+Natural+Language+Processing+and+Speech+Tools+for+Italian%2E)
6. Final Workshop (EVALITA 2018), Turin, Italy. CEUR.org. [find](https://scholar.google.com/scholar?q=Final+Workshop+%28EVALITA+2018%29%2C+Turin%2C+Italy%2E+CEUR%2Eorg%2E)
7. Pierpaolo Basile and Nicole Novielli. 2014. UNIBA at EVALITA 2014-SENTIPOLC Task: Predicting tweet sentiment polarity combining micro-blogging, lexicon and semantic features. In Proceedings of Fourth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. [find](https://scholar.google.com/scholar?q=Pierpaolo+Basile+and+Nicole+Novielli%2E+2014%2E+UNIBA+at+EVALITA+2014%2DSENTIPOLC+Task%3A+Predicting+tweet+sentiment+polarity+combining+micro%2Dblogging%2C+lexicon+and+semantic+features%2E+In+Proceedings+of+Fourth+)
8. Final Workshop (EVALITA 2014). CEUR.org. [find](https://scholar.google.com/scholar?q=Final+Workshop+%28EVALITA+2014%29%2E+CEUR%2Eorg%2E)
9. Angelo Basile and Chiara Rubagotti. 2018. Automatic Identification of Misogyny in English and Italian Tweets at EVALITA 2018 with a Multilingual Hate Lexicon. In Proceedings of Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. [find](https://scholar.google.com/scholar?q=Angelo+Basile+and+Chiara+Rubagotti%2E+2018%2E+Automatic+Identification+of+Misogyny+in+English+and+Italian+Tweets+at+EVALITA+2018+with+a+Multilingual+Hate+Lexicon%2E+In+Proceedings+of+Sixth+Evaluation+Campai)
10. Final Workshop (EVALITA 2018), Turin, Italy. CEUR.org. [find](https://scholar.google.com/scholar?q=Final+Workshop+%28EVALITA+2018%29%2C+Turin%2C+Italy%2E+CEUR%2Eorg%2E)
11. Davide Buscaldi. 2018. Tweetaneuse AMI EVALITA2018: Character-based Models for the Automatic Misogyny Identification Task. In Proceedings of Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. [find](https://scholar.google.com/scholar?q=Davide+Buscaldi%2E+2018%2E+Tweetaneuse+AMI+EVALITA2018%3A+Character%2Dbased+Models+for+the+Automatic+Misogyny+Identification+Task%2E+In+Proceedings+of+Sixth+Evaluation+Campaign+of+Natural+Language+Processing+an)
12. Final Workshop (EVALITA 2018), Turin, Italy. CEUR.org. [find](https://scholar.google.com/scholar?q=Final+Workshop+%28EVALITA+2018%29%2C+Turin%2C+Italy%2E+CEUR%2Eorg%2E)
13. R.G. Capuano. 2007. Turpia: sociologia del turpil-oquio e della bestemmia. Riscontri (Milan, Italy). Costa & Nolan. [find](https://scholar.google.com/scholar?q=R%2EG%2E+Capuano%2E+2007%2E+Turpia%3A+sociologia+del+turpil%2Doquio+e+della+bestemmia%2E+Riscontri+%28Milan%2C+Italy%29%2E+Costa+%26+Nolan%2E)
14. Tommaso Caselli, Nicole Novielli, Viviana Patti, and Paolo Rosso. 2018. [find](https://scholar.google.com/scholar?q=Tommaso+Caselli%2C+Nicole+Novielli%2C+Viviana+Patti%2C+and+Paolo+Rosso%2E+2018%2E)
15. EVALITA 2018: Overview of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. In Tommaso Caselli, Nicole Novielli, Viviana Patti, and Paolo Rosso, editors, Proceedings of Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. [find](https://scholar.google.com/scholar?q=EVALITA+2018%3A+Overview+of+the+6th+Evaluation+Campaign+of+Natural+Language+Processing+and+Speech+Tools+for+Italian%2E+In+Tommaso+Caselli%2C+Nicole+Novielli%2C+Viviana+Patti%2C+and+Paolo+Rosso%2C+editors%2C+Proceed)
16. Final Workshop (EVALITA 2018), Turin, Italy. CEUR.org. [find](https://scholar.google.com/scholar?q=Final+Workshop+%28EVALITA+2018%29%2C+Turin%2C+Italy%2E+CEUR%2Eorg%2E)
17. Maeve Duggan. 2017. Online Harassment. [http://www.pewinternet.org/2017/](http://www.pewinternet.org/2017/) 07/11/online-harassment-2017/. Last accessed 2018-10-28. [↗](http://www.pewinternet.org/2017/)
18. Fabio Fasoli, Andrea Carnaghi, and Maria Paola Paladino. 2015. Social acceptability of sexist derogatory and sexist objectifying slurs across contexts. Language Sciences, 52:98–107. [find](https://scholar.google.com/scholar?q=Fabio+Fasoli%2C+Andrea+Carnaghi%2C+and+Maria+Paola+Paladino%2E+2015%2E+Social+acceptability+of+sexist+derogatory+and+sexist+objectifying+slurs+across+contexts%2E+Language+Sciences%2C+52%3A98%E2%80%93107%2E)
19. Elisabetta Fersini, M Anzovino, and P Rosso. 2018. Overview of the task on automatic misogyny identification at ibereval. In Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018). CEUR Workshop Proceedings. CEUR-WS. org, Seville, Spain. [find](https://scholar.google.com/scholar?q=Elisabetta+Fersini%2C+M+Anzovino%2C+and+P+Rosso%2E+2018%2E+Overview+of+the+task+on+automatic+misogyny+identification+at+ibereval%2E+In+Proceedings+of+the+Third+Workshop+on+Evaluation+of+Human+Language+Technolog)
20. Paula Fortuna, Ilaria Bonavita, and Sérgio Nunes. 2018. INESC TEC, Eurecat and Porto University. [find](https://scholar.google.com/scholar?q=Paula+Fortuna%2C+Ilaria+Bonavita%2C+and+Se%CC%81rgio+Nunes%2E+2018%2E+INESC+TEC%2C+Eurecat+and+Porto+University%2E)
21. Simona Frenda, Ghanem Bilal, et al. 2018a. Exploration of Misogyny in Spanish and English tweets. In Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018), volume 2150, pages 260–267. Ceur Workshop Proceedings. [find](https://scholar.google.com/scholar?q=Simona+Frenda%2C+Ghanem+Bilal%2C+et+al%2E+2018a%2E+Exploration+of+Misogyny+in+Spanish+and+English+tweets%2E+In+Third+Workshop+on+Evaluation+of+Human+Language+Technologies+for+Iberian+Languages+%28IberEval+2018%29%2C+)
22. Simona Frenda, Bilal Ghanem, Estefanı́a Guzmán-Falcón, Manuel Montes-y-Gómez, and Luis Vil-laseñor-Pineda. 2018b. Automatic Lexicons Expansion for Multilingual Misogyny Detection. In Proceedings of Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. [find](https://scholar.google.com/scholar?q=Simona+Frenda%2C+Bilal+Ghanem%2C+Estefan%C4%B1%CC%81a+Guzma%CC%81n%2DFalco%CC%81n%2C+Manuel+Montes%2Dy%2DGo%CC%81mez%2C+and+Luis+Vil%2Dlasen%CC%83or%2DPineda%2E+2018b%2E+Automatic+Lexicons+Expansion+for+Multilingual+Misogyny+Detection%2E+In+Proceedings+o)
23. Final Workshop (EVALITA 2018), Turin, Italy. CEUR.org. [find](https://scholar.google.com/scholar?q=Final+Workshop+%28EVALITA+2018%29%2C+Turin%2C+Italy%2E+CEUR%2Eorg%2E)
24. Rob Goot, Nikola Ljubešić, Ian Matroos, Malvina Nissim, and Barbara Plank. 2018. Bleaching Text: Abstract Features for Cross-lingual Gender Prediction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 383–389. [find](https://scholar.google.com/scholar?q=Rob+Goot%2C+Nikola+Ljubes%CC%8Cic%CC%81%2C+Ian+Matroos%2C+Malvina+Nissim%2C+and+Barbara+Plank%2E+2018%2E+Bleaching+Text%3A+Abstract+Features+for+Cross%2Dlingual+Gender+Prediction%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+t)
25. Sarah Hewitt, Thanassis Tiropanis, and Christian Bokhove. 2016. The Problem of identifying Misogynist Language on Twitter (and other online social spaces). In Proceedings of the 8th ACM Conference on Web Science, pages 333–335. ACM. [find](https://scholar.google.com/scholar?q=Sarah+Hewitt%2C+Thanassis+Tiropanis%2C+and+Christian+Bokhove%2E+2016%2E+The+Problem+of+identifying+Misogynist+Language+on+Twitter+%28and+other+online+social+spaces%29%2E+In+Proceedings+of+the+8th+ACM+Conference+on+)
26. Endang Wahyu Pamungkas, Alessandra Teresa Cignarella, Valerio Basile, and Viviana Patti. 2018. Automatic Identification of Misogyny in English and Italian Tweets at EVALITA 2018 with a Multilingual Hate Lexicon. In Proceedings of Sixth Evaluation Campaign of Natural Language Processing and Speech To… [find](https://scholar.google.com/scholar?q=Endang+Wahyu+Pamungkas%2C+Alessandra+Teresa+Cignarella%2C+Valerio+Basile%2C+and+Viviana+Patti%2E+2018%2E+Automatic+Identification+of+Misogyny+in+English+and+Italian+Tweets+at+EVALITA+2018+with+a+Multilingual+Ha)
27. Final Workshop (EVALITA 2018), Turin, Italy. CEUR.org. [find](https://scholar.google.com/scholar?q=Final+Workshop+%28EVALITA+2018%29%2C+Turin%2C+Italy%2E+CEUR%2Eorg%2E)
28. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on Empirical Methods in Natural Language Processing, pages 1532–1543. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+2014+conference+on+Empirical+Methods+in+Natural+Language+Process)
29. Bailey Poland. 2016. Haters: Harassment, Abuse, and Violence Online. Potomac Books, Incorporated. [find](https://scholar.google.com/scholar?q=Bailey+Poland%2E+2016%2E+Haters%3A+Harassment%2C+Abuse%2C+and+Violence+Online%2E+Potomac+Books%2C+Incorporated%2E)
30. Punyajoy Saha, Binny Mathew, Pawan Goyal, and Animesh Mukherjee. 2018. Indian Institute of Engineering Science and Technology (Shibpur), Indian Institute of Technology (Kharagpur). [find](https://scholar.google.com/scholar?q=Punyajoy+Saha%2C+Binny+Mathew%2C+Pawan+Goyal%2C+and+Animesh+Mukherjee%2E+2018%2E+Indian+Institute+of+Engineering+Science+and+Technology+%28Shibpur%29%2C+Indian+Institute+of+Technology+%28Kharagpur%29%2E)
31. Elena Shushkevich and John Cardiff. 2018. Misogyny detection and classification in English tweets. In Proceedings of Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. [find](https://scholar.google.com/scholar?q=Elena+Shushkevich+and+John+Cardiff%2E+2018%2E+Misogyny+detection+and+classification+in+English+tweets%2E+In+Proceedings+of+Sixth+Evaluation+Campaign+of+Natural+Language+Processing+and+Speech+Tools+for+Itali)
32. Final Workshop (EVALITA 2018), Turin, Italy. CEUR.org. [find](https://scholar.google.com/scholar?q=Final+Workshop+%28EVALITA+2018%29%2C+Turin%2C+Italy%2E+CEUR%2Eorg%2E)