---
title: "CH-SIMS — paper by Yu et al. (2020) — summary and ideas"
description: "As a baseline, they proposed a multi-task framework for late-fusion models. 21 references, cited by 53 papers here."
authors: ["Wenmeng Yu", "Hua Xu", "Fanyang Meng", "Yilin Zhu", "Yixiao Ma", "Jiele Wu", "Jiyun Zou", "Kaicheng Yang"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.343/"
doi: "10.18653/v1/2020.acl-main.343"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yu-2020-ch-sims-chinese-multimodal-sentiment/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yu-2020-ch-sims-chinese-multimodal-sentiment/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/) / CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality

# CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality

**Authors:** [Wenmeng Yu](https://md.research.elamur.ai/free/en/authors/wenmeng-yu/), [Hua Xu](https://md.research.elamur.ai/free/en/authors/hua-xu/), Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou and Kaicheng Yang · **Year:** 2020 · **Venue:** [Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2020](https://md.research.elamur.ai/free/en/venues/acl/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.343/) · [PDF](https://aclanthology.org/2020.acl-main.343.pdf) · [DOI 10.18653/v1/2020.acl-main.343](https://doi.org/10.18653/v1/2020.acl-main.343) · [OpenAlex](https://openalex.org/W3034849760)

**Citations:** 440 citations (OpenAlex, 2026-10-14) · [53](https://md.research.elamur.ai/free/en/papers/yu-2020-ch-sims-chinese-multimodal-sentiment/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary

Existing multimodal sentiment datasets give each video segment a single shared label. That label may not match what text, voice, or face convey individually, and it pushes late-fusion models toward overly similar per-modality representations. To address this, the authors built CH-SIMS, a Mandarin dataset of short clips from movies, TV series, and variety shows. Each clip carries a multimodal sentiment label plus separate labels for text, audio, and vision. Annotators judged each modality in isolation, in the order text, audio, silent video, and finally the full clip.

As a baseline, they proposed a multi-task framework for late-fusion models. Modality-specific feature sub-networks in the lower layers are shared between the unimodal tasks and the main multimodal task. They plugged TFN, LMF, and LF-DNN into this framework. Adding the unimodal objectives improved multimodal prediction and yielded more distinct per-modality representations. Ablations showed the full set of auxiliary tasks works best, while partial subsets help little or even hurt. The resource supports both single-modality and fused sentiment research.

## Key points

- CH-SIMS contains 2,281 segments cut from 60 raw videos, featuring 474 distinct speakers. Segments last 1–10 seconds, averaging 3.67 s and about 15 words.
- Five annotators labeled each clip as −1, 0, or 1 per modality. Averaging the scores gives 11 regression values, which are also grouped into 5 classes (negative to positive). Negative clips outnumber positive ones.
- Measured label disagreement was smallest between audio and multimodal labels and largest between vision and text.
- With MLF-DNN, using all three unimodal auxiliary tasks raised Acc-2 from 80.04 to 82.28 and F1 from 80.40 to 82.52. It also lowered MAE from 43.95 to 40.64 (as reported in %). The authors attribute weaker partial combinations to learning asynchrony across subtasks.
- Data and code are released at github.com/thuiar/MMSA.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 137 works · 83,262 citations
- [State Key Laboratory of Intelligent Technology and Systems](https://md.research.elamur.ai/free/en/labs/state-key-laboratory-of-intelligent-technology-and-systems/) — Research institute, Beijing, China · 21 works · 9,641 citations
- [Beijing National Research Center for Information Science and Technology](https://md.research.elamur.ai/free/en/labs/beijing-national-research-center-for-information-science-and-technology/) — Research institute, Beijing, China · 21 works · 5,843 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 798 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 463 works

🧭 Topics (2): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 609 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 269 works

## Cite

Cite as: Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou and Kaicheng Yang. 2020. CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yu2020ch,
  title = {{CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality}},
  author = {Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou and Kaicheng Yang},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.343},
  url = {https://doi.org/10.18653/v1/2020.acl-main.343}
}
```

- [← previous FLAT: Chinese NER Using Flat-Lattice Transformer](https://md.research.elamur.ai/free/en/papers/li-2020-flat-chinese-ner-flat-lattice/)
- [next → DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://md.research.elamur.ai/free/en/papers/xin-2020-deebert-dynamic-early-exiting-accelerating/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/)

## References

21 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Wu Liu, Tao Mei, Yongdong Zhang, Cherry Che, and Jiebo Luo. 2015. Multi-task deep visual-semantic embedding for video thumbnail selection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3707–3715. [find](https://scholar.google.com/scholar?q=Wu+Liu%2C+Tao+Mei%2C+Yongdong+Zhang%2C+Cherry+Che%2C+and+Jiebo+Luo%2E+2015%2E+Multi%2Dtask+deep+visual%2Dsemantic+embedding+for+video+thumbnail+selection%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+)
2. Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2018. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2):423–443. [find](https://scholar.google.com/scholar?q=Tadas+Baltrus%CC%8Caitis%2C+Chaitanya+Ahuja%2C+and+Louis%2DPhilippe+Morency%2E+2018%2E+Multimodal+machine+learning%3A+A+survey+and+taxonomy%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%2C+41%282%29%3A423%E2%80%9344)
3. Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. 2018. [Efficient low-rank multimodal fusion with modality-specific factors](https://md.research.elamur.ai/free/en/papers/liu-2018-efficient-low-rank-multimodal-fusion/). arXiv preprint arXiv:1806.00064.
4. Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. 2018. Openface 2.0: Facial behavior analysis toolkit. In 2018 13th IEEE Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579–2605. [find](https://scholar.google.com/scholar?q=Tadas+Baltrusaitis%2C+Amir+Zadeh%2C+Yao+Chong+Lim%2C+and+Louis%2DPhilippe+Morency%2E+2018%2E+Openface+2%2E0%3A+Facial+behavior+analysis+toolkit%2E+In+2018+13th+IEEE+Laurens+van+der+Maaten+and+Geoffrey+Hinton%2E+2008%2E+Vis)
5. Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. 2015. librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science conference, volume 8. [find](https://scholar.google.com/scholar?q=Brian+McFee%2C+Colin+Raffel%2C+Dawen+Liang%2C+Daniel+PW+Ellis%2C+Matt+McVicar%2C+Eric+Battenberg%2C+and+Oriol+Nieto%2E+2015%2E+librosa%3A+Audio+and+music+signal+analysis+in+python%2E+In+Proceedings+of+the+14th+python+in+)
6. Louis-Philippe Morency, Rada Mihalcea, and Payal Doshi. 2011. Towards multimodal sentiment analysis: Harvesting opinions from the web. In Proceedings of the 13th international conference on multimodal interfaces, pages 169–176. ACM. [find](https://scholar.google.com/scholar?q=Louis%2DPhilippe+Morency%2C+Rada+Mihalcea%2C+and+Payal+Doshi%2E+2011%2E+Towards+multimodal+sentiment+analysis%3A+Harvesting+opinions+from+the+web%2E+In+Proceedings+of+the+13th+international+conference+on+multimodal)
7. Bo Pang, Lillian Lee, et al. 2008. Opinion mining and sentiment analysis. Foundations and Trends R in Information Retrieval, 2(1–2):1–135. [find](https://scholar.google.com/scholar?q=Bo+Pang%2C+Lillian+Lee%2C+et+al%2E+2008%2E+Opinion+mining+and+sentiment+analysis%2E+Foundations+and+Trends+R+in+Information+Retrieval%2C+2%281%E2%80%932%29%3A1%E2%80%93135%2E)
8. Verónica Pérez-Rosas, Rada Mihalcea, and Louis-Philippe Morency. 2013. Utterance-level multimodal sentiment analysis. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 973–982. [find](https://scholar.google.com/scholar?q=Vero%CC%81nica+Pe%CC%81rez%2DRosas%2C+Rada+Mihalcea%2C+and+Louis%2DPhilippe+Morency%2E+2013%2E+Utterance%2Dlevel+multimodal+sentiment+analysis%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+Association+for+Computational+L)
9. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Mazumder, Amir Zadeh, and Louis-Philippe Morency. 2017. Multi-level multiple attentions for contextual multimodal sentiment analysis. In 2017 IEEE International Conference on Data Mining (ICDM), pages 1033–1038. IEEE. [find](https://scholar.google.com/scholar?q=Soujanya+Poria%2C+Erik+Cambria%2C+Devamanyu+Hazarika%2C+Navonil+Mazumder%2C+Amir+Zadeh%2C+and+Louis%2DPhilippe+Morency%2E+2017%2E+Multi%2Dlevel+multiple+attentions+for+contextual+multimodal+sentiment+analysis%2E+In+2017+)
10. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2018. [Meld: A multimodal multi-party dataset for emotion recognition in conversations](https://md.research.elamur.ai/free/en/papers/poria-2018-meld-multimodal-multi-party-dataset/). arXiv preprint arXiv:1810.02508.
11. Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. [Multimodal transformer for unaligned multimodal language sequences](https://md.research.elamur.ai/free/en/papers/tsai-2019-multimodal-transformer-unaligned-multimodal-language/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6558–6569, F…
12. Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. [Tensor fusion network for multimodal sentiment analysis](https://md.research.elamur.ai/free/en/papers/zadeh-2017-tensor-fusion-network-multimodal-sentiment/). arXiv preprint arXiv:1707.07250.
13. Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018a. Memory fusion network for multi-view sequential learning. In Thirty-Second AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Amir+Zadeh%2C+Paul+Pu+Liang%2C+Navonil+Mazumder%2C+Soujanya+Poria%2C+Erik+Cambria%2C+and+Louis%2DPhilippe+Morency%2E+2018a%2E+Memory+fusion+network+for+multi%2Dview+sequential+learning%2E+In+Thirty%2DSecond+AAAI+Conference)
14. Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259. [find](https://scholar.google.com/scholar?q=Amir+Zadeh%2C+Rowan+Zellers%2C+Eli+Pincus%2C+and+Louis%2DPhilippe+Morency%2E+2016%2E+Mosi%3A+multimodal+corpus+of+sentiment+intensity+and+subjectivity+analysis+in+online+opinion+videos%2E+arXiv+preprint+arXiv%3A1606%2E06)
15. AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018b. [Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph](https://md.research.elamur.ai/free/en/papers/zadeh-2018-multimodal-language-analysis-wild-cmu/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume…
16. Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. 2016a. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 23(10):1499–1503. [find](https://scholar.google.com/scholar?q=Kaipeng+Zhang%2C+Zhanpeng+Zhang%2C+Zhifeng+Li%2C+and+Yu+Qiao%2E+2016a%2E+Joint+face+detection+and+alignment+using+multitask+cascaded+convolutional+networks%2E+IEEE+Signal+Processing+Letters%2C+23%2810%29%3A1499%E2%80%931503%2E)
17. Wenlu Zhang, Rongjian Li, Tao Zeng, Qian Sun, Sudhir Kumar, Jieping Ye, and Shuiwang Ji. 2016b. Deep model based transfer and multi-task learning for biological image analysis. IEEE transactions on Big Data. [find](https://scholar.google.com/scholar?q=Wenlu+Zhang%2C+Rongjian+Li%2C+Tao+Zeng%2C+Qian+Sun%2C+Sudhir+Kumar%2C+Jieping+Ye%2C+and+Shuiwang+Ji%2E+2016b%2E+Deep+model+based+transfer+and+multi%2Dtask+learning+for+biological+image+analysis%2E+IEEE+transactions+on+Bi)
18. Yu Zhang and Qiang Yang. 2017. A survey on multi-task learning. arXiv preprint arXiv:1707.08114. — [arXiv:1707.08114](https://arxiv.org/abs/1707.08114) [find](https://scholar.google.com/scholar?q=Yu+Zhang+and+Qiang+Yang%2E+2017%2E+A+survey+on+multi%2Dtask+learning%2E+arXiv+preprint+arXiv%3A1707%2E08114%2E+%E2%80%94+%5BarXiv%3A1707%2E08114%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1707%2E08114%29)
19. Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2018. Learning factorized multimodal representations. [find](https://scholar.google.com/scholar?q=Yao%2DHung+Hubert+Tsai%2C+Paul+Pu+Liang%2C+Amir+Zadeh%2C+Louis%2DPhilippe+Morency%2C+and+Ruslan+Salakhutdinov%2E+2018%2E+Learning+factorized+multimodal+representations%2E)
20. Jennifer Williams, Steven Kleinegesse, Ramona Comanescu, and Oana Radu. 2018. Recognizing emotions in video using multimodal DNN feature fusion. In Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML), pages 11–19, Melbourne, Australia. Association for Computation… [find](https://scholar.google.com/scholar?q=Jennifer+Williams%2C+Steven+Kleinegesse%2C+Ramona+Comanescu%2C+and+Oana+Radu%2E+2018%2E+Recognizing+emotions+in+video+using+multimodal+DNN+feature+fusion%2E+In+Proceedings+of+Grand+Challenge+and+Workshop+on+Human)
21. Martin Wöllmer, Felix Weninger, Tobias Knaup, Björn Schuller, Congkai Sun, Kenji Sagae, and Louis-Philippe Morency. 2013. Youtube movie reviews: Sentiment analysis in an audio-visual context. IEEE Intelligent Systems, 28(3):46–53. [find](https://scholar.google.com/scholar?q=Martin+Wo%CC%88llmer%2C+Felix+Weninger%2C+Tobias+Knaup%2C+Bjo%CC%88rn+Schuller%2C+Congkai+Sun%2C+Kenji+Sagae%2C+and+Louis%2DPhilippe+Morency%2E+2013%2E+Youtube+movie+reviews%3A+Sentiment+analysis+in+an+audio%2Dvisual+context%2E+IEEE+I)

*Updated 24 October 2026*