---
title: "A Challenge Dataset and Effective Models for — Jiang 2019"
description: "Standard aspect-based sentiment analysis (ABSA) benchmarks have a hidden weakness."
authors: ["Qingnan Jiang", "Lei Chen", "Ruifeng Xu", "Xiang Ao", "Min Yang"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1654/"
doi: "10.18653/v1/d19-1654"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jiang-2019-challenge-dataset-effective-models-aspect/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jiang-2019-challenge-dataset-effective-models-aspect/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/) / A Challenge Dataset and Effective Models for Aspect-Based Sentiment Analysis

# A Challenge Dataset and Effective Models for Aspect-Based Sentiment Analysis

**Authors:** Qingnan Jiang, Lei Chen, [Ruifeng Xu](https://md.research.elamur.ai/free/en/authors/ruifeng-xu/), Xiang Ao, [Min Yang](https://md.research.elamur.ai/free/en/authors/min-yang/) · **Institutions:** [Harbin Institute of Technology](https://md.research.elamur.ai/free/en/universities/harbin-institute-of-technology/) · [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) · [Institute of Computing Technology](https://md.research.elamur.ai/free/en/labs/institute-of-computing-technology/) · [Shenzhen Institutes of Advanced Technology](https://md.research.elamur.ai/free/en/labs/shenzhen-institutes-of-advanced-technology/) · [Peng Cheng Laboratory](https://md.research.elamur.ai/free/en/labs/peng-cheng-laboratory/) · **Year:** 2019 · **Venue:** Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1654/) · [PDF](https://aclanthology.org/D19-1654.pdf) · [DOI 10.18653/v1/d19-1654](https://doi.org/10.18653/v1/d19-1654) · [OpenAlex](https://openalex.org/W2971014768)

**Citations:** 306 citations (OpenAlex, 2026-10-14) · [87](https://md.research.elamur.ai/free/en/papers/jiang-2019-challenge-dataset-effective-models-aspect/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Standard aspect-based sentiment analysis (ABSA) benchmarks have a hidden weakness. In most of their sentences, either one aspect appears or all aspects share the same polarity. A model can therefore score well by predicting overall sentence sentiment and ignoring the target aspect. To fix this, the authors built MAMS (Multi-Aspect Multi-Sentiment) from Citysearch New York restaurant reviews. Every sentence in it mentions at least two aspects carrying different polarities. MAMS has versions for both aspect-term (ATSA) and aspect-category (ACSA) sentiment analysis. They also propose CapsNet, a capsule-network classifier with two additions: an aspect-aware normalization step, and a routing scheme guided by sentiment capsules initialized from sentiment-word embeddings. CapsNet-BERT is a variant that uses pre-trained BERT as its encoder. On MAMS, aspect-agnostic classifiers fall to roughly 50% accuracy, and several attention-based ABSA systems lose much of their advantage. CapsNet-BERT achieves the best scores across all tested datasets. MAMS thus serves as a stricter test of whether models truly link sentiment to specific aspects.

## Key points
- MAMS contains 13,854 ATSA instances and 8,879 ACSA instances. That is about 2.87× and 1.87× the size of SemEval-2014 Restaurant. Sentences average 2.62 aspect terms and 2.25 aspect categories, and the ACSA version uses eight categories.
- In existing datasets, few sentences mix aspects with different polarities: 26.58% (Restaurant-ATSA), 9.58% (Restaurant-ACSA), 20.05% (Laptop) and 0.09% (Twitter). A MAMS-small split matches the Restaurant training size for fair comparison.
- TextCNN reaches about 76% on Restaurant ATSA but only about 53% on MAMS ATSA. MemNet and AEN, which do not model word order, also perform poorly.
- CapsNet-BERT outperforms vanilla BERT on MAMS: 83.39 vs. 82.22 (ATSA) and 79.46 vs. 78.29 (ACSA).
- Replacing capsule-guided routing with standard dynamic routing (the -DR variants) lowers results on every dataset. Data and code are available at github.com/siat-nlp/MAMS-for-ABSA.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{jiang2019challenge,
  title = {{A Challenge Dataset and Effective Models for Aspect-Based Sentiment Analysis}},
  author = {Jiang, Qingnan and Chen, Lei and Xu, Ruifeng and Ao, Xiang and Yang, Min},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1654},
  url = {https://doi.org/10.18653/v1/d19-1654}
}
```

- [← previous Disentangled Representation Learning for Non-Parallel Text Style Transfer](https://md.research.elamur.ai/free/en/papers/john-2019-disentangled-representation-learning-non-parallel/)
- [next → Cross-lingual Transfer Learning for Multilingual Task Oriented Dialog](https://md.research.elamur.ai/free/en/papers/schuster-2019-cross-lingual-transfer-learning-multilingual/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/)

## References

20 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Langua… [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+van+Merrienboer%2C+Caglar+Gulcehre%2C+Dzmitry+Bahdanau%2C+Fethi+Bougares%2C+Holger+Schwenk%2C+and+Yoshua+Bengio%2E+2014%2E+Learning+phrase+representations+using+rnn+encoder%E2%80%93decoder+for+statistic)
2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2E)
3. Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. 2014. Adaptive recursive neural network for target-dependent twitter sentiment classification. In Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 2: Short papers), volume 2, pages 49–54. [find](https://scholar.google.com/scholar?q=Li+Dong%2C+Furu+Wei%2C+Chuanqi+Tan%2C+Duyu+Tang%2C+Ming+Zhou%2C+and+Ke+Xu%2E+2014%2E+Adaptive+recursive+neural+network+for+target%2Ddependent+twitter+sentiment+classification%2E+In+Proceedings+of+the+52nd+annual+meetin) [our card](https://md.research.elamur.ai/free/en/papers/dong-2014-adaptive-recursive-neural-network-target/)
4. Gayatree Ganu, Nomie Elhadad, and Amlie Marian. 2009. Beyond the stars: Improving rating predictions using review text content. In WebDB. [find](https://scholar.google.com/scholar?q=Gayatree+Ganu%2C+Nomie+Elhadad%2C+and+Amlie+Marian%2E+2009%2E+Beyond+the+stars%3A+Improving+rating+predictions+using+review+text+content%2E+In+WebDB%2E)
5. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770– 778. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+2016%2E+Deep+residual+learning+for+image+recognition%2E+In+2016+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+pages+770%E2%80%93+778%2E)
6. Binxuan Huang, Yanglan Ou, and Kathleen M Carley. 2018. Aspect level sentiment classification with attention-over-attention neural networks. In International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation, pages 197–… [find](https://scholar.google.com/scholar?q=Binxuan+Huang%2C+Yanglan+Ou%2C+and+Kathleen+M+Carley%2E+2018%2E+Aspect+level+sentiment+classification+with+attention%2Dover%2Dattention+neural+networks%2E+In+International+Conference+on+Social+Computing%2C+Behavioral)
7. Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: A method for stochastic optimization. international conference on learning representations. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Lei+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+international+conference+on+learning+representations%2E)
8. Zeyang Lei, Yujiu Yang, Min Yang, Wei Zhao, Jun Guo, and Yi Liu. 2019. A human-like semantic cognition network for aspect-level sentiment classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6650–6657. [find](https://scholar.google.com/scholar?q=Zeyang+Lei%2C+Yujiu+Yang%2C+Min+Yang%2C+Wei+Zhao%2C+Jun+Guo%2C+and+Yi+Liu%2E+2019%2E+A+human%2Dlike+semantic+cognition+network+for+aspect%2Dlevel+sentiment+classification%2E+In+Proceedings+of+the+AAAI+Conference+on+Artif)
9. Suresh Manandhar. 2014. Semeval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 27–35. [find](https://scholar.google.com/scholar?q=Suresh+Manandhar%2E+2014%2E+Semeval%2D2014+task+4%3A+Aspect+based+sentiment+analysis%2E+In+Proceedings+of+the+8th+International+Workshop+on+Semantic+Evaluation+%28SemEval+2014%29%2C+pages+27%E2%80%9335%2E) [our card](https://md.research.elamur.ai/free/en/papers/pontiki-2014-semeval-2014-task-4-aspect/)
10. Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. 2017. Dynamic routing between capsules. neural information processing systems, pages 3856–3866. [find](https://scholar.google.com/scholar?q=Sara+Sabour%2C+Nicholas+Frosst%2C+and+Geoffrey+E+Hinton%2E+2017%2E+Dynamic+routing+between+capsules%2E+neural+information+processing+systems%2C+pages+3856%E2%80%933866%2E)
11. Youwei Song, Jiahai Wang, Tao Jiang, Zhiyue Liu, and Yanghui Rao. 2019. Attentional encoder network for targeted sentiment classification. arXiv preprint arXiv:1902.09314. — [arXiv:1902.09314](https://arxiv.org/abs/1902.09314) [find](https://scholar.google.com/scholar?q=Youwei+Song%2C+Jiahai+Wang%2C+Tao+Jiang%2C+Zhiyue+Liu%2C+and+Yanghui+Rao%2E+2019%2E+Attentional+encoder+network+for+targeted+sentiment+classification%2E+arXiv+preprint+arXiv%3A1902%2E09314%2E+%E2%80%94+%5BarXiv%3A1902%2E09314%5D%28https%3A%2F)
12. Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2016a. Effective lstms for target-dependent sentiment classification. international conference on computational linguistics, pages 3298–3307. [find](https://scholar.google.com/scholar?q=Duyu+Tang%2C+Bing+Qin%2C+Xiaocheng+Feng%2C+and+Ting+Liu%2E+2016a%2E+Effective+lstms+for+target%2Ddependent+sentiment+classification%2E+international+conference+on+computational+linguistics%2C+pages+3298%E2%80%933307%2E) [our card](https://md.research.elamur.ai/free/en/papers/tang-2016-effective-lstms-target-dependent-sentiment/)
13. Duyu Tang, Bing Qin, and Ting Liu. 2016b. Aspect level sentiment classification with deep memory network. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 214–224. [find](https://scholar.google.com/scholar?q=Duyu+Tang%2C+Bing+Qin%2C+and+Ting+Liu%2E+2016b%2E+Aspect+level+sentiment+classification+with+deep+memory+network%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pag) [our card](https://md.research.elamur.ai/free/en/papers/tang-2016-aspect-level-sentiment-classification-deep/)
14. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. neural information processing systems, pages 5998–6008. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+neural+information+processing+systems%2C+pa)
15. Yequan Wang, Minlie Huang, xiaoyan zhu, and Li Zhao. 2016. Attention-based lstm for aspect-level sentiment classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 606–615. [find](https://scholar.google.com/scholar?q=Yequan+Wang%2C+Minlie+Huang%2C+xiaoyan+zhu%2C+and+Li+Zhao%2E+2016%2E+Attention%2Dbased+lstm+for+aspect%2Dlevel+sentiment+classification%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Languag) [our card](https://md.research.elamur.ai/free/en/papers/wang-2016-attention-based-lstm-aspect-level/)
16. Wei Xue and Tao Li. 2018. Aspect based sentiment analysis with gated convolutional networks. meeting of the association for computational linguistics, 1:2514–2523. [find](https://scholar.google.com/scholar?q=Wei+Xue+and+Tao+Li%2E+2018%2E+Aspect+based+sentiment+analysis+with+gated+convolutional+networks%2E+meeting+of+the+association+for+computational+linguistics%2C+1%3A2514%E2%80%932523%2E)
17. Min Yang, Qingnan Jiang, Ying Shen, Qingyao Wu, Zhou Zhao, and Wei Zhou. 2019. Hierarchical human-like strategy for aspect-level sentiment classification with sentiment linguistic knowledge and reinforcement learning. Neural Networks. [find](https://scholar.google.com/scholar?q=Min+Yang%2C+Qingnan+Jiang%2C+Ying+Shen%2C+Qingyao+Wu%2C+Zhou+Zhao%2C+and+Wei+Zhou%2E+2019%2E+Hierarchical+human%2Dlike+strategy+for+aspect%2Dlevel+sentiment+classification+with+sentiment+linguistic+knowledge+and+reinfo)
18. Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2017. Interactive attention networks for aspect-level sentiment classification. In IJCAI’17 Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 4068–4074. [find](https://scholar.google.com/scholar?q=Dehong+Ma%2C+Sujian+Li%2C+Xiaodong+Zhang%2C+and+Houfeng+Wang%2E+2017%2E+Interactive+attention+networks+for+aspect%2Dlevel+sentiment+classification%2E+In+IJCAI%E2%80%9917+Proceedings+of+the+26th+International+Joint+Conferen)
19. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+D%2E+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+2014+Conference+on+Empirical+Methods+in+Natural+Language+Proc)
20. Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and [find](https://scholar.google.com/scholar?q=Maria+Pontiki%2C+Dimitris+Galanis%2C+John+Pavlopoulos%2C+Harris+Papageorgiou%2C+Ion+Androutsopoulos%2C+and)