--- title: "SemEval-2019 Task 3 — Chatterjee 2019" description: "EmoContext was a SemEval-2019 shared task on recognizing emotion in short textual conversations." authors: ["Ankush Chatterjee", "Kedhar Nath Narahari", "Meghana Joshi", "Puneet Agrawal"] year: 2019 venue: "Proceedings of the 13th International Workshop on Semantic Evaluation" source: "https://aclanthology.org/S19-2005/" doi: "10.18653/v1/s19-2005" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/chatterjee-2019-semeval-2019-task-3-emocontext/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chatterjee-2019-semeval-2019-task-3-emocontext/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / SemEval-2019 Task 3: EmoContext Contextual Emotion Detection in Text # SemEval-2019 Task 3: EmoContext Contextual Emotion Detection in Text **Authors:** Ankush Chatterjee, Kedhar Nath Narahari, Meghana Joshi, Puneet Agrawal · **Year:** 2019 · **Venue:** Proceedings of the 13th International Workshop on Semantic Evaluation · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/S19-2005/) · [PDF](https://aclanthology.org/S19-2005.pdf) · [DOI 10.18653/v1/s19-2005](https://doi.org/10.18653/v1/s19-2005) · [OpenAlex](https://openalex.org/W2955429306) **Citations:** 283 citations (openalex-snapshot · 2026-10-14) · [126](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary EmoContext was a SemEval-2019 shared task on recognizing emotion in short textual conversations. Text lacks facial and vocal cues, and the meaning of a reply often depends on what came before it. Each instance is a three-turn exchange: a user message, a response from a conversational agent, and a second user message. Systems must label the final turn as Happy, Sad, Angry or Others. The organizers mined about 2 million dialogues from a year of user interactions with a conversational agent. They removed offensive content, personally identifiable information and non-English text. To find emotional dialogues without labelling huge random samples, they used embedding-based similarity search before human annotation. They released a training set and two evaluation sets. In the first phase 311 teams took part, and 164 continued to the final phase. The winning system scored 0.7959 micro-averaged F1, against a baseline of 0.5861. The resource gives a shared benchmark for context-aware emotion detection in informal chat, which matters for empathetic dialogue agents. ## Key points - **Data sizes:** Train has 30160 dialogues, Test1 has 2755 and Test2 has 5509. The final ranking used Test2. - **Class balance:** - Train: Others make up roughly half (14948), and the rest are split among Happy (4243), Sad (5463) and Angry (5506). - Test sets: Others make up about 85%, which mirrors how rarely emotions actually appear in chat. - **Annotator agreement:** Kappa was 0.58 on training data and 0.59 on evaluation data. The authors read this as evidence that people genuinely perceive the emotion of a conversation differently. - **Leaderboard:** The top teams, Leo1020 and Mfzszgs, did not publish system papers. Only 46 teams submitted descriptions overall. Across all participants the median F1µ was 0.694. - **Common techniques:** - BiLSTM/LSTM architectures with attention. - GloVe embeddings; BERT, ELMo and ULMFiT for transfer learning; DeepMoji and Emoji2Vec for emoticons. - The ekphrasis package for preprocessing. - Ensembles were nearly universal. - Most systems did best on Sad and worst on Happy. ## Interesting ideas - **Seed-and-expand annotation:** The organizers started from about 300 labelled examples per class. They embedded these, retrieved similar unlabelled dialogues with a cosine-similarity threshold, and pruned the candidates with heuristics (for example, a crying emoticon in a Happy candidate) before sending them to human judges. This cut annotation effort roughly fivefold and suits any rare-class labelling project. - **Realistic test distribution:** The evaluation sets were deliberately skewed toward Others, unlike the more balanced training set. This tests how systems behave under real-world class sparsity and rewards methods that handle distribution shift, as SNU IDS explicitly did. - **Context-flipping examples:** Some dialogues were included where the last turn alone suggests one emotion but the full context implies another, such as "I started crying" after good news. This is a cheap, convincing way to show and probe why context modelling matters. - **Staged evaluation design:** Test1 labels were released mid-competition, and the daily submission limit dropped from 20 to 10 in the final phase. The authors also suggest fully blind evaluation in future, where participants never see the test inputs. This is worth considering for any benchmark exposed to leaderboard overfitting. - **Decomposing the label space:** THU-HCSI trained separate models for the full four-way task, for the three emotions only, and for Others versus emotion. It then combined them by voting. This is a simple route to robustness when one class dominates. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 126 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 122 not yet. - [HeBERT & HebEMO: a Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition](https://md.research.elamur.ai/free/en/papers/langstream-2021-chriqui-hebert-hebemo-hebrew-bert-model/) - [Directed Acyclic Graph Network for Conversational Emotion Recognition](https://md.research.elamur.ai/free/en/papers/shen-2021-directed-acyclic-graph-network-conversational/) - [Emotion Transcription in Conversation: A Benchmark for Capturing Subtle and Complex Emotional States through Natural Language](https://md.research.elamur.ai/free/en/papers/tanaka-2026-emotion-transcription-conversation-benchmark-capturing/) - [Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning](https://md.research.elamur.ai/free/en/papers/wang-2023-label-words-anchors-information-flow/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{chatterjee2019semeval, title = {{SemEval-2019 Task 3: EmoContext Contextual Emotion Detection in Text}}, author = {Chatterjee, Ankush and Narahari, Kedhar Nath and Joshi, Meghana and Agrawal, Puneet}, booktitle = {Proceedings of the 13th International Workshop on Semantic Evaluation}, year = {2019}, doi = {10.18653/v1/s19-2005}, url = {https://doi.org/10.18653/v1/s19-2005} } ``` - [← previous Semantic Noise Matters for Neural Natural Language Generation](https://md.research.elamur.ai/free/en/papers/dusek-2019-semantic-noise-matters-neural-natural/) - [next → SemEval-2019 Task 4: Hyperpartisan News Detection](https://md.research.elamur.ai/free/en/papers/kiesel-2019-semeval-2019-task-4-hyperpartisan/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) ## References 2 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Muhammad Abdul-Mageed and Lyle Ungar. 2017. Emonet: Fine-grained emotion detection with gated recurrent neural networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, volume Vol. 1, pages 718–728. [find](https://scholar.google.com/scholar?q=Muhammad+Abdul%2DMageed+and+Lyle+Ungar%2E+2017%2E+Emonet%3A+Fine%2Dgrained+emotion+detection+with+gated+recurrent+neural+networks%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+) [our card](https://md.research.elamur.ai/free/en/papers/mageed-2017-emonet-fine-grained-emotion-detection/) 2. Cecilia Ovesdotter Alm, Dan Roth, and Richard Sproat. 2005. Emotions from text: machine learning for text-based emotion prediction. In Proceedings of the conference on human language technology and empirical methods in natural language processing, pages 579–586. ACL. [find](https://scholar.google.com/scholar?q=Cecilia+Ovesdotter+Alm%2C+Dan+Roth%2C+and+Richard+Sproat%2E+2005%2E+Emotions+from+text%3A+machine+learning+for+text%2Dbased+emotion+prediction%2E+In+Proceedings+of+the+conference+on+human+language+technology+and+em) [our card](https://md.research.elamur.ai/free/en/papers/alm-2005-emotions-text-machine-learning-text/)