---
title: "Locate and Explain — paper by Wan et al. (2026)"
description: "Studies of why speakers feel emotions in multimodal dialogue usually take one of two approaches. 30 references."
authors: ["Jikun Wan", "Chen Gong", "Guohong Fu"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.2012/"
doi: "10.18653/v1/2026.acl-long.2012"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wan-2026-locate-explain-joint-multimodal-emotion/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wan-2026-locate-explain-joint-multimodal-emotion/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/35/) / Locate and Explain: Joint Multimodal Emotion Cause Extraction and Summarization in Conversation

# Locate and Explain: Joint Multimodal Emotion Cause Extraction and Summarization in Conversation

**Authors:** Jikun Wan, [Chen Gong](https://md.research.elamur.ai/free/en/authors/chen-gong/) and [Guohong Fu](https://md.research.elamur.ai/free/en/authors/guohong-fu/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.2012/) · [PDF](https://aclanthology.org/2026.acl-long.2012.pdf) · [DOI 10.18653/v1/2026.acl-long.2012](https://doi.org/10.18653/v1/2026.acl-long.2012) · [OpenAlex](https://openalex.org/W7166843240)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Studies of why speakers feel emotions in multimodal dialogue usually take one of two approaches. Some pick out the utterances that trigger an emotion, which shows where the cause is but not what it is. Others write free-text cause descriptions, which explain the cause but cannot be traced to specific turns. The authors combine both in a new task, MECES. A model must select the cause utterances and also write an abstractive summary of the cause.

To support the task, they built MECESD from Chinese TV-series dialogues in M3ED. Annotation followed Ellis's ABC theory from psychology, which treats an emotion as the product of an activating event and the speaker's belief together. Gemini-2.5-Pro produced draft labels, and human annotators then refined them.

They also propose MPF-LLM, an end-to-end LLM model that learns both subtasks jointly. Its Multi-Level Fusion (MLF) module brings audio and visual features into the model. Joint training helped both subtasks, which suggests that locating causes and explaining them reinforce each other.

## Key points
- **Dataset size:** MECESD covers 781 cleaned dialogues (10,136 utterances) taken from the 811-dialogue M3ED evaluation release. It contains 5,787 emotion utterances with causes, 12,231 emotion-cause pairs and 6,040 cause summaries. One emotion may have several valid summaries.
- **Annotation scheme:** Cause labels capture both the triggering event (A) and the speaker's belief (B), not just a single factor. Before annotation, fragmented utterances were merged by meaning and multimodal timestamps were realigned.
- **Joint vs. extraction-only:** On MECESD, jointly trained MPF-LLM reaches 0.7948 extraction F1, compared with 0.7879 for an extraction-only variant. In ablations, removing MLF or the audio/visual inputs mostly lowers scores.
- **Human evaluation:** On 50 test samples rated 0–5, MPF-LLM's overall score was 3.40. ChatGLM3-6B scored 3.24 and 3-shot DeepSeek-V3.2 scored 3.16. Every system rated clearly lower on beliefs than on events, so inferring speakers' beliefs remains an open problem.
- **Release:** Data and code are planned for release at github.com/wwwper/MECES, restricted to non-commercial academic use.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Soochow University](https://md.research.elamur.ai/free/en/universities/soochow-university/) — University, Suzhou, China · 63 works · 4,507 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 870 works

🧭 Topics (3): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 972 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works · [📝 Summarization](https://md.research.elamur.ai/free/en/topics/summarization/) · 232 works

## Cite

Cite as: Jikun Wan, Chen Gong and Guohong Fu. 2026. Locate and Explain: Joint Multimodal Emotion Cause Extraction and Summarization in Conversation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wan2026locate,
  title = {{Locate and Explain: Joint Multimodal Emotion Cause Extraction and Summarization in Conversation}},
  author = {Jikun Wan, Chen Gong and Guohong Fu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2012},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2012}
}
```

- [← previous LLMSurgeon: Diagnosing Data Mixture of Large Language Models](https://md.research.elamur.ai/free/en/papers/luo-2026-llmsurgeon-diagnosing-data-mixture-large/)
- [next → LocRegen: Cost-Efficient Redundancy Removal in Multilingual E-commerce Titles with Small Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2026-locregen-cost-efficient-redundancy-removal/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/35/)

## References

30 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. [Qwen2.5-vl technical report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen2-5-vl-technical/). Preprint, arXiv:2502.13…
2. Satanjeev Banerjee and Alon Lavie. 2005. [METEOR: An automatic metric for MT evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Ass…
3. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. [find](https://scholar.google.com/scholar?q=Jacob+Cohen%2E+1960%2E+A+coefficient+of+agreement+for+nominal+scales%2E+Educational+and+psychological+measurement%2C+20%281%29%3A37%E2%80%9346%2E)
4. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, and 3416 [other…](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/)
5. DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenhao Xu, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, and 245 others. 2025. [Deepseek-v3.2: Pushing the frontier of open…](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-v3-2-pushing/)
6. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. [GLM: General language model pretraining with autoregressive blank infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335…
7. Paul Ekman. 1992. [An argument for basic emotions](https://md.research.elamur.ai/free/en/papers/ekman-1992-argument-basic-emotions/). Cognition & emotion, 6(3-4):169–200.
8. Albert Ellis. 1957. Rational psychotherapy and individual psychology. Journal of individual psychology, 13(1):38. [find](https://scholar.google.com/scholar?q=Albert+Ellis%2E+1957%2E+Rational+psychotherapy+and+individual+psychology%2E+Journal+of+individual+psychology%2C+13%281%29%3A38%2E)
9. Hao Fei, Han Zhang, Bin Wang, Lizi Liao, Qian Liu, and Erik Cambria. 2024. EmpathyEar: An open-source avatar multimodal empathetic chatbot. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 61–71, Bangkok, Thailand. As… [find](https://scholar.google.com/scholar?q=Hao+Fei%2C+Han+Zhang%2C+Bin+Wang%2C+Lizi+Liao%2C+Qian+Liu%2C+and+Erik+Cambria%2E+2024%2E+EmpathyEar%3A+An+open%2Dsource+avatar+multimodal+empathetic+chatbot%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Association)
10. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. [Hubert: Self-supervised speech representation learning by masked prediction of hidden units](https://md.research.elamur.ai/free/en/papers/hsu-2021-hubert-self-supervised-speech-representation/). IEEE/ACM transactions on audio, speech, and language processing, 29:3451–3460.
11. Guimin Hu, Zhihong Zhu, Daniel Hershcovich, Lijie Hu, Hasti Seifi, and Jiayuan Xie. 2024. UniMEEC: Towards unified multimodal emotion recognition and emotion cause. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 5248–5261, Miami, Florida, USA. Association for Computa… [find](https://scholar.google.com/scholar?q=Guimin+Hu%2C+Zhihong+Zhu%2C+Daniel+Hershcovich%2C+Lijie+Hu%2C+Hasti+Seifi%2C+and+Jiayuan+Xie%2E+2024%2E+UniMEEC%3A+Towards+unified+multimodal+emotion+recognition+and+emotion+cause%2E+In+Findings+of+the+Association+for+)
12. Bobo Li, Hao Fei, Fei Li, Tat-seng Chua, and Donghong Ji. 2024. Multimodal emotion-cause pair extraction with holistic interaction and label constraint. ACM Trans. Multimedia Comput. Commun. Appl. Just Accepted. [find](https://scholar.google.com/scholar?q=Bobo+Li%2C+Hao+Fei%2C+Fei+Li%2C+Tat%2Dseng+Chua%2C+and+Donghong+Ji%2E+2024%2E+Multimodal+emotion%2Dcause+pair+extraction+with+holistic+interaction+and+label+constraint%2E+ACM+Trans%2E+Multimedia+Comput%2E+Commun%2E+Appl%2E+Jus)
13. Zheng Lian, Haoyu Chen, Lan Chen, Haiyang Sun, Li-cai Sun, Yong Ren, Zebang Cheng, Bin Liu, Rui Liu, Xiaojiang Peng, Jiangyan Yi, and Jianhua Tao. 2025. Affectgpt: A new dataset, model, and benchmark for emotion understanding with multimodal large language models. Preprint, arXiv:2501.16566. — [arXiv:2501.16566](https://arxiv.org/abs/2501.16566) [find](https://scholar.google.com/scholar?q=Zheng+Lian%2C+Haoyu+Chen%2C+Lan+Chen%2C+Haiyang+Sun%2C+Li%2Dcai+Sun%2C+Yong+Ren%2C+Zebang+Cheng%2C+Bin+Liu%2C+Rui+Liu%2C+Xiaojiang+Peng%2C+Jiangyan+Yi%2C+and+Jianhua+Tao%2E+2025%2E+Affectgpt%3A+A+new+dataset%2C+model%2C+and+benchmark+)
14. Qiao Liang, Ying Shen, Tiantian Chen, and Lin Zhang. 2025. M3HG: Multimodal, multi-scale, and multi-type node heterogeneous graph for emotion cause triplet extraction in conversations. In Findings of the Association for Computational Linguistics: ACL 2025, pages 11416–11431, Vienna, Austria. Associa… [find](https://scholar.google.com/scholar?q=Qiao+Liang%2C+Ying+Shen%2C+Tiantian+Chen%2C+and+Lin+Zhang%2E+2025%2E+M3HG%3A+Multimodal%2C+multi%2Dscale%2C+and+multi%2Dtype+node+heterogeneous+graph+for+emotion+cause+triplet+extraction+in+conversations%2E+In+Findings+of+)
15. Chin-Yew Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pages 74–81.
16. OpenAI. 2025. Introducing GPT-5. Accessed: 2025-12-18. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025%2E+Introducing+GPT%2D5%2E+Accessed%3A+2025%2D12%2D18%2E)
17. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
18. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In International conference on machine learning, pages 8748–8…
19. Nils Reimers and Iryna Gurevych. 2019. [Sentence-BERT: Sentence embeddings using Siamese BERT-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
20. Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. CoRR, abs/1706.09799. [find](https://scholar.google.com/scholar?q=Shikhar+Sharma%2C+Layla+El+Asri%2C+Hannes+Schulz%2C+and+Jeremie+Zumer%2E+2017%2E+Relevance+of+unsupervised+metrics+in+task%2Doriented+dialogue+for+evaluating+natural+language+generation%2E+CoRR%2C+abs%2F1706%2E09799%2E)
21. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
22. Fanfan Wang, Zixiang Ding, Rui Xia, Zhaoyu Li, and Jianfei Yu. 2023. Multimodal emotion-cause pair extraction in conversations. IEEE Transactions on Affective Computing, 14(3):1832–1844. [find](https://scholar.google.com/scholar?q=Fanfan+Wang%2C+Zixiang+Ding%2C+Rui+Xia%2C+Zhaoyu+Li%2C+and+Jianfei+Yu%2E+2023%2E+Multimodal+emotion%2Dcause+pair+extraction+in+conversations%2E+IEEE+Transactions+on+Affective+Computing%2C+14%283%29%3A1832%E2%80%931844%2E)
23. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 23 others. 2025a. Qwen2.5 technical report. Preprint, arXiv:24… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+Yang%2C+Jin)
24. Yang Yang, Xunde Dong, and Yupeng Qiang. 2025b. Mse-adapter: a lightweight plugin endowing llms with the capability to perform multimodal sentiment analysis and emotion recognition. In Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovat… [find](https://scholar.google.com/scholar?q=Yang+Yang%2C+Xunde+Dong%2C+and+Yupeng+Qiang%2E+2025b%2E+Mse%2Dadapter%3A+a+lightweight+plugin+endowing+llms+with+the+capability+to+perform+multimodal+sentiment+analysis+and+emotion+recognition%2E+In+Proceedings+of+)
25. Duzhen Zhang, Zhen Yang, Fandong Meng, Xiuyi Chen, and Jie Zhou. 2022. TSAM: A two-stream attention model for causal emotion entailment. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6762–6772, Gyeongju, Republic of Korea. International Committee on Computat… [find](https://scholar.google.com/scholar?q=Duzhen+Zhang%2C+Zhen+Yang%2C+Fandong+Meng%2C+Xiuyi+Chen%2C+and+Jie+Zhou%2E+2022%2E+TSAM%3A+A+two%2Dstream+attention+model+for+causal+emotion+entailment%2E+In+Proceedings+of+the+29th+International+Conference+on+Computat)
26. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+In+International+Conference+on+Learning+Representations%2E)
27. Jinming Zhao, Tenggan Zhang, Jingwen Hu, Yuchen Liu, Qin Jin, Xinchao Wang, and Haizhou Li. 2022. M3ED: Multi-modal multi-scene multi-label emotional dialogue database. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5699– 571… [find](https://scholar.google.com/scholar?q=Jinming+Zhao%2C+Tenggan+Zhang%2C+Jingwen+Hu%2C+Yuchen+Liu%2C+Qin+Jin%2C+Xinchao+Wang%2C+and+Haizhou+Li%2E+2022%2E+M3ED%3A+Multi%2Dmodal+multi%2Dscene+multi%2Dlabel+emotional+dialogue+database%2E+In+Proceedings+of+the+60th+Annu)
28. Fanfan Wang, Heqing Ma, Xiangqing Shen, Jianfei Yu, and Rui Xia. 2024. Observe before generate: Emotion-cause aware video caption for multimodal emotion cause generation in conversations. In Proceedings of the 32nd ACM International Conference on Multimedia, MM ’24, page 5820–5828, New York, NY, USA… [find](https://scholar.google.com/scholar?q=Fanfan+Wang%2C+Heqing+Ma%2C+Xiangqing+Shen%2C+Jianfei+Yu%2C+and+Rui+Xia%2E+2024%2E+Observe+before+generate%3A+Emotion%2Dcause+aware+video+caption+for+multimodal+emotion+cause+generation+in+conversations%2E+In+Proceedin)
29. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, and 1 others. 2025. [Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency](https://md.research.elamur.ai/free/en/papers/wang-2025-internvl3-5-advancing-open/). arXiv preprint arXiv:2508.18265.
30. Dan Wu, Xincheng Ju, Dong Zhang, Shoushan Li, Erik Cambria, and Guodong Zhou. 2025. Emotion across modalities and cultures: Multilingual multimodal emotion-cause analysis with memory-inspired framework. In Proceedings of the 33rd ACM International Conference on Multimedia, MM ’25, page 5775–5783, Ne… [find](https://scholar.google.com/scholar?q=Dan+Wu%2C+Xincheng+Ju%2C+Dong+Zhang%2C+Shoushan+Li%2C+Erik+Cambria%2C+and+Guodong+Zhou%2E+2025%2E+Emotion+across+modalities+and+cultures%3A+Multilingual+multimodal+emotion%2Dcause+analysis+with+memory%2Dinspired+framewor)

*Updated 1 November 2026*