---
title: "The Multilingual Amazon Reviews Corpus — Keung 2020"
description: "Multilingual text classification lacks large, openly available benchmarks."
authors: ["Phillip Keung", "Yichao Lu", "György Szarvas", "Noah A. Smith"]
year: 2020
venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/2020.emnlp-main.369/"
doi: "10.18653/v1/2020.emnlp-main.369"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/keung-2020-multilingual-amazon-reviews-corpus/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/keung-2020-multilingual-amazon-reviews-corpus/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) / The Multilingual Amazon Reviews Corpus

# The Multilingual Amazon Reviews Corpus

**Authors:** [Phillip Keung](https://md.research.elamur.ai/free/en/authors/phillip-keung/), [Yichao Lu](https://md.research.elamur.ai/free/en/authors/yichao-lu/), [György Szarvas](https://md.research.elamur.ai/free/en/authors/gyorgy-szarvas/), [Noah A. Smith](https://md.research.elamur.ai/free/en/authors/noah-a-smith/) · **Institutions:** [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) · [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) · [Amazon](https://md.research.elamur.ai/free/en/companies/amazon/) · **Year:** 2020 · **Venue:** Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-main.369/) · [PDF](https://aclanthology.org/2020.emnlp-main.369.pdf) · [DOI 10.18653/v1/2020.emnlp-main.369](https://doi.org/10.18653/v1/2020.emnlp-main.369) · [OpenAlex](https://openalex.org/W3092547334)

**Citations:** 146 citations (OpenAlex, 2026-10-14) · [109](https://md.research.elamur.ai/free/en/papers/keung-2020-multilingual-amazon-reviews-corpus/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Multilingual text classification lacks large, openly available benchmarks. Existing options such as RCV1/RCV2, MLDoc, XNLI, Yelp and earlier Amazon review releases are small, hard to obtain, missing non-English training data, unreliable about language labels, or lacking fixed splits. The authors address this gap with the Multilingual Amazon Reviews Corpus (MARC). It covers six languages (English, Japanese, German, French, Spanish and Chinese), with reviews collected from six national Amazon marketplaces and posted between late 2015 and late 2019. The data was filtered by language identification, vocabulary frequency and sampling caps. It was also balanced so that every star rating forms one fifth of each language. Each language has fixed train, development and test splits.

As baselines, they fine-tuned cased mBERT for both in-language and zero-shot cross-lingual rating prediction. Because star ratings are ordered, they argue for mean absolute error (MAE) as the main metric rather than accuracy. Adding the title and product category consistently lowers error. Transfer to other languages works but remains clearly weaker than in-language training. The corpus is distributed via AWS Open Datasets for non-commercial research.

## Key points
- **Size and splits:** 200,000 training, 5,000 development and 5,000 test reviews per language. Each record holds the review body, title, stars, anonymized reviewer and product IDs, and a product category (30 common types plus "other").
- **Filtering:**
  - Only verified purchases were kept.
  - At most 20 reviews were taken per product or reviewer.
  - Reviews needed at least 20 characters and were truncated at 2,000.
  - fastText-style language ID (threshold 0.8) was applied; manual checks found only 1 misclassified review in 1,200.
  - Reviews containing rare tokens were dropped.
- **Supervised baselines (mBERT base, 110M parameters):** average MAE×100 falls from 54.9 to 48.2 for 5-star prediction when title and category are added. Binarized prediction falls from 9.1 to 6.9. Accuracy rises from 54.4% to 59.2%.
- **Zero-shot transfer from English:** non-English targets average 76.9 MAE×100 (about 44% accuracy). More English training data helps, but with diminishing returns. Checkpoints were selected using target-language development sets.
- **Category skew:** product category distributions differ across languages, and Chinese reviews are dominated by books.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{keung2020multilingual,
  title = {{The Multilingual Amazon Reviews Corpus}},
  author = {Keung, Phillip and Lu, Yichao and Szarvas, György and Smith, Noah A.},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.369},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.369}
}
```

- [← previous Re-evaluating Evaluation in Text Summarization](https://md.research.elamur.ai/free/en/papers/bhandari-2020-re-evaluating-evaluation-text-summarization/)
- [next → Decolonising Speech and Language Technology](https://md.research.elamur.ai/free/en/papers/bird-2020-decolonising-speech-language-technology/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/)

## References

22 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amazon Inc. 2015. Amazon customer reviews dataset. [https://registry.opendata.aws/](https://registry.opendata.aws/) amazon-reviews/ . [↗](https://registry.opendata.aws/)
2. Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597–610. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe+and+Holger+Schwenk%2E+2019%2E+Massively+multilingual+sentence+embeddings+for+zero%2Dshot+cross%2Dlingual+transfer+and+beyond%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%3A597) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2018-massively-multilingual-sentence-embeddings-zero/)
3. Nuria Bel, Cornelis HA Koster, and Marta Villegas. 2003. Cross-lingual text categorization. In International Conference on Theory and Practice of Digital Libraries, pages 126–139. Springer. [find](https://scholar.google.com/scholar?q=Nuria+Bel%2C+Cornelis+HA+Koster%2C+and+Marta+Villegas%2E+2003%2E+Cross%2Dlingual+text+categorization%2E+In+International+Conference+on+Theory+and+Practice+of+Digital+Libraries%2C+pages+126%E2%80%93139%2E+Springer%2E)
4. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. [find](https://scholar.google.com/scholar?q=Piotr+Bojanowski%2C+Edouard+Grave%2C+Armand+Joulin%2C+and+Tomas+Mikolov%2E+2017%2E+Enriching+word+vectors+with+subword+information%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+5%3A135%E2%80%93146%2E) [our card](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/)
5. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. text ca… [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirica)
6. Yichao Lu, Phillip Keung, Faisal Ladhak, Vikas Bhardwaj, Shaonan Zhang, and Jason Sun. 2018. A neural interlingua for multilingual machine translation. WMT. [find](https://scholar.google.com/scholar?q=Yichao+Lu%2C+Phillip+Keung%2C+Faisal+Ladhak%2C+Vikas+Bhardwaj%2C+Shaonan+Zhang%2C+and+Jason+Sun%2E+2018%2E+A+neural+interlingua+for+multilingual+machine+translation%2E+WMT%2E) [our card](https://md.research.elamur.ai/free/en/papers/lu-2018-neural-interlingua-multilingual-machine-translation/)
7. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon… [find](https://scholar.google.com/scholar?q=Andrew+L%2E+Maas%2C+Raymond+E%2E+Daly%2C+Peter+T%2E+Pham%2C+Dan+Huang%2C+Andrew+Y%2E+Ng%2C+and+Christopher+Potts%2E+2011%2E+Learning+word+vectors+for+sentiment+analysis%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Ass) [our card](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/)
8. Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (… [find](https://scholar.google.com/scholar?q=Jianmo+Ni%2C+Jiacheng+Li%2C+and+Julian+McAuley%2E+2019%2E+Justifying+recommendations+using+distantly%2Dlabeled+reviews+and+fine%2Dgrained+aspects%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Nat) [our card](https://md.research.elamur.ai/free/en/papers/ni-2019-justifying-recommendations-distantly-labeled-reviews/)
9. Alexander Pak and Patrick Paroubek. 2010. Twitter as a corpus for sentiment analysis and opinion mining. In LREC, volume 10, pages 1320–1326. [find](https://scholar.google.com/scholar?q=Alexander+Pak+and+Patrick+Paroubek%2E+2010%2E+Twitter+as+a+corpus+for+sentiment+analysis+and+opinion+mining%2E+In+LREC%2C+volume+10%2C+pages+1320%E2%80%931326%2E) [our card](https://md.research.elamur.ai/free/en/papers/pak-2010-twitter-as-corpus-sentiment-analysis/)
10. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzma%CC%81n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Unsupervised+cross%2Dling)
11. Peter Prettenhofer and Benno Stein. 2010. Cross-language text classification using structural correspondence learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1118–1127, Uppsala, Sweden. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Peter+Prettenhofer+and+Benno+Stein%2E+2010%2E+Cross%2Dlanguage+text+classification+using+structural+correspondence+learning%2E+In+Proceedings+of+the+48th+Annual+Meeting+of+the+Association+for+Computational+Li)
12. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computation… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Adina+Williams%2C+Samuel+R%2E+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+Xnli%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+)
13. Reuters Ltd. 2005. Reuters corpus, volume 2, multilingual corpus. [https://trec.nist.gov/data/](https://trec.nist.gov/data/) reuters/reuters.html . [↗](https://trec.nist.gov/data/)
14. Gerard De Melo and Stefan Siersdorfer. 2007. Multilingual text classification using ontologies. In European Conference on Information Retrieval, pages 541–548. Springer. [find](https://scholar.google.com/scholar?q=Gerard+De+Melo+and+Stefan+Siersdorfer%2E+2007%2E+Multilingual+text+classification+using+ontologies%2E+In+European+Conference+on+Information+Retrieval%2C+pages+541%E2%80%93548%2E+Springer%2E)
15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+NAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
16. Phillip Keung, Yichao Lu, and Vikas Bhardwaj. 2019. Adversarial learning with contextual embeddings for zero-resource cross-lingual classification and ner. arXiv preprint arXiv:1909.00153. [find](https://scholar.google.com/scholar?q=Phillip+Keung%2C+Yichao+Lu%2C+and+Vikas+Bhardwaj%2E+2019%2E+Adversarial+learning+with+contextual+embeddings+for+zero%2Dresource+cross%2Dlingual+classification+and+ner%2E+arXiv+preprint+arXiv%3A1909%2E00153%2E)
17. Phillip Keung, Yichao Lu, Julian Salazar, and Vikas Bhardwaj. 2020. On the evaluation of contextual embeddings for zero-shot cross-lingual transfer learning. ArXiv, abs/2004.15001. — [arXiv:2004.15001](https://arxiv.org/abs/2004.15001) [find](https://scholar.google.com/scholar?q=Phillip+Keung%2C+Yichao+Lu%2C+Julian+Salazar%2C+and+Vikas+Bhardwaj%2E+2020%2E+On+the+evaluation+of+contextual+embeddings+for+zero%2Dshot+cross%2Dlingual+transfer+learning%2E+ArXiv%2C+abs%2F2004%2E15001%2E+%E2%80%94+%5BarXiv%3A2004%2E15001)
18. Holger Schwenk and Xian Li. 2018. A corpus for multilingual document classification in eight languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Holger+Schwenk+and+Xian+Li%2E+2018%2E+A+corpus+for+multilingual+document+classification+in+eight+languages%2E+In+Proceedings+of+the+Eleventh+International+Conference+on+Language+Resources+and+Evaluation%2C+LR) [our card](https://md.research.elamur.ai/free/en/papers/schwenk-2018-corpus-multilingual-document-classification-eight/)
19. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of bert. EMNLP. [find](https://scholar.google.com/scholar?q=Shijie+Wu+and+Mark+Dredze%2E+2019%2E+Beto%2C+bentz%2C+becas%3A+The+surprising+cross%2Dlingual+effectiveness+of+bert%2E+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/wu-2019-beto-bentz-becas-surprising-cross/)
20. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 14… [find](https://scholar.google.com/scholar?q=Zichao+Yang%2C+Diyi+Yang%2C+Chris+Dyer%2C+Xiaodong+He%2C+Alex+Smola%2C+and+Eduard+Hovy%2E+2016%2E+Hierarchical+attention+networks+for+document+classification%2E+In+Proceedings+of+the+2016+conference+of+the+North+Amer) [our card](https://md.research.elamur.ai/free/en/papers/yang-2016-hierarchical-attention-networks-document-classification/)
21. Yelp Inc. 2019. Yelp open dataset. [https://www.yelp.com/dataset](https://www.yelp.com/dataset) . [↗](https://www.yelp.com/dataset)
22. David D Lewis, Yiming Yang, Tony G Rose, and Fan Li. 2004. Rcv1: A new benchmark collection for [find](https://scholar.google.com/scholar?q=David+D+Lewis%2C+Yiming+Yang%2C+Tony+G+Rose%2C+and+Fan+Li%2E+2004%2E+Rcv1%3A+A+new+benchmark+collection+for)