--- title: "Language Models are Few-shot Multilingual — Winata 2021" description: "The paper asks whether large language models trained mostly on English can perform multi-class intent classification in other languages through…" authors: ["Genta Indra Winata", "Andrea Madotto", "Zhaojiang Lin", "Rosanne Liu", "Jason Yosinski", "Pascale Fung"] year: 2021 venue: "Proceedings of the 1st Workshop on Multilingual Representation Learning" source: "https://aclanthology.org/2021.mrl-1.1/" doi: "10.18653/v1/2021.mrl-1.1" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/winata-2021-language-models-are-few-shot/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/winata-2021-language-models-are-few-shot/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / Language Models are Few-shot Multilingual Learners # Language Models are Few-shot Multilingual Learners **Authors:** [Genta Indra Winata](https://md.research.elamur.ai/free/en/authors/genta-indra-winata/), [Andrea Madotto](https://md.research.elamur.ai/free/en/authors/andrea-madotto/), [Zhaojiang Lin](https://md.research.elamur.ai/free/en/authors/zhaojiang-lin/), [Rosanne Liu](https://md.research.elamur.ai/free/en/authors/rosanne-liu/), [Jason Yosinski](https://md.research.elamur.ai/free/en/authors/jason-yosinski/), [Pascale Fung](https://md.research.elamur.ai/free/en/authors/pascale-fung/) · **Year:** 2021 · **Venue:** Proceedings of the 1st Workshop on Multilingual Representation Learning · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2021.mrl-1.1/) · [PDF](https://aclanthology.org/2021.mrl-1.1.pdf) · [DOI 10.18653/v1/2021.mrl-1.1](https://doi.org/10.18653/v1/2021.mrl-1.1) · [OpenAlex](https://openalex.org/W3199377785) **Citations:** 93 citations (openalex-snapshot · 2026-10-14) · [80](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary The paper asks whether large language models trained mostly on English can perform multi-class intent classification in other languages through in-context learning alone, with no parameter updates. The authors test GPT-2, GPT NEO, GPT NEO-J and T5 on SNIPS, MTOP (English, French, German, Spanish) and MultiNLU (English, Spanish). They examine both monolingual prompts and a cross-lingual setup, where the English examples in the context differ from the language of the query. Their method turns each intent class into a separate binary question. The prompt holds positive and negative examples for that class, and the model's normalized probability of answering "true" serves as a confidence score. The class with the highest score wins. Accuracy rises well above chance as more shots are added and as model size grows. GPT models do better than T5. In the cross-lingual MultiNLU setting, GPT NEO-J nearly matches fine-tuned and translation-based systems. This suggests that gradient-free prompting is a cheap option for cross-lingual NLU when labeled data or compute is scarce. ## Key points - **Data:** SNIPS (English), MTOP (en/fr/de/es) and MultiNLU (en/es). Results are reported as mean accuracy with standard deviation over three runs. - **Models and limits:** GPT-2, GPT NEO, GPT NEO-J and T5 were evaluated. Context limits shaped the prompt design: 1,024 tokens for GPT-2 XL and 2,048 for GPT NEO. - **Cross-lingual result:** On MultiNLU (en→es), GPT NEO-J trails the fine-tuned state-of-the-art baseline of Liu et al. (2020b) by about 4.2%. It trails the Translate-Train model by under 3%. - **Language pairs:** On MTOP, en→es transfer generally works better than en→de or en→fr. T5 lags behind GPT models, possibly because of how it was pre-trained. - **Baselines and code:** The baselines were zero-shot, zero-shot cross-task (entailment-style scoring) and fine-tuning. Code is available at github.com/gentaiscool/few-shot-lm. ## Interesting ideas - **Per-class confidence scoring instead of free generation:** Each class gets its own binary true/false query, and the normalized probability of "true" is compared across classes. This avoids the model producing unexpected words. It could fit any task with a closed label set. - **One prompt per class:** Building a separate short prompt for each label avoids packing every class's examples into one context window. The prompts can also be run in parallel. This is helpful when there are many labels or the context length is limited. - **Prompt in one language, query in another:** Using an English context with a target-language query is a cheap probe of cross-lingual ability. It could be used to check how well a mostly-English model handles another language before investing in annotation. - **Order the examples within the prompt:** Placing positive examples before negative ones gave better results than random shuffling, especially for T5. Measuring variance across shuffled orders is a simple robustness check for any prompting setup. - **Study how token overlap affects transfer:** The authors suggest correlating transfer quality with token and named-entity overlap between source and target languages. They note that English tokenizers handle non-Latin scripts poorly. This analysis is directly relevant when judging whether such models would work for languages written in Cyrillic or other scripts. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 80 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 70 not yet. - [Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study](https://md.research.elamur.ai/free/en/papers/cao-2023-assessing-cross-cultural-alignment-between/) - [Bridging the Resource Gap: Exploring the Efficacy of English and Multilingual LLMs for Swedish](https://md.research.elamur.ai/free/en/papers/holmstrom-2023-bridging-resource-gap-exploring-efficacy/) - [Few-shot Learning with Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/) - [Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages](https://md.research.elamur.ai/free/en/papers/qin-2023-cross-lingual-prompting-improving-zero/) - [Language Models are Multilingual Chain-of-Thought Reasoners](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/) - [mGPT: Few-Shot Learners Go Multilingual](https://md.research.elamur.ai/free/en/papers/shliazhko-2022-mgpt-few-shot-learners-go/) - [TyDiP: A Dataset for Politeness Classification in Nine Typologically Diverse Languages](https://md.research.elamur.ai/free/en/papers/srinivasan-2022-tydip-dataset-politeness-classification-nine/) - [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/) - [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/) - [Ethical and social risks of harm from Language Models](https://md.research.elamur.ai/free/en/papers/weidinger-2021-ethical-social-risks-harm-language/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{winata2021language, title = {{Language Models are Few-shot Multilingual Learners}}, author = {Winata, Genta Indra and Madotto, Andrea and Lin, Zhaojiang and Liu, Rosanne and Yosinski, Jason and Fung, Pascale}, booktitle = {Proceedings of the 1st Workshop on Multilingual Representation Learning}, year = {2021}, doi = {10.18653/v1/2021.mrl-1.1}, url = {https://doi.org/10.18653/v1/2021.mrl-1.1} } ``` - [← previous LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://md.research.elamur.ai/free/en/papers/schuhmann-2021-laion-400m-open-dataset-clip/) - [next → Large Dual Encoders Are Generalizable Retrievers](https://md.research.elamur.ai/free/en/papers/ni-2021-large-dual-encoders-generalizable-retrievers/) - [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) ## References 3 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Gustavo Aguilar, Bryan McCann, Tong Niu, Nazneen Rajani, N. Keskar, and T. Solorio. 2020. Char2subword: Extending the subword embedding space from pre-trained models using robust character compositionality. ArXiv, abs/2010.12730. — [arXiv:2010.12730](https://arxiv.org/abs/2010.12730) [find](https://scholar.google.com/scholar?q=Gustavo+Aguilar%2C+Bryan+McCann%2C+Tong+Niu%2C+Nazneen+Rajani%2C+N%2E+Keskar%2C+and+T%2E+Solorio%2E+2020%2E+Char2subword%3A+Extending+the+subword+embedding+space+from+pre%2Dtrained+models+using+robust+character+composition) 2. Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597–610. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe+and+Holger+Schwenk%2E+2019%2E+Massively+multilingual+sentence+embeddings+for+zero%2Dshot+cross%2Dlingual+transfer+and+beyond%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%3A597) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2018-massively-multilingual-sentence-embeddings-zero/) 3. Abhik Bhattacharjee, Tahmid Hasan, Kazi Samin, M Sohel Rahman, Anindya Iqbal, and Rifat Shahriyar. 2021. Banglabert: Combating embedding barrier for low-resource language understanding. arXiv preprint arXiv:2101.00204. — [arXiv:2101.00204](https://arxiv.org/abs/2101.00204) [find](https://scholar.google.com/scholar?q=Abhik+Bhattacharjee%2C+Tahmid+Hasan%2C+Kazi+Samin%2C+M+Sohel+Rahman%2C+Anindya+Iqbal%2C+and+Rifat+Shahriyar%2E+2021%2E+Banglabert%3A+Combating+embedding+barrier+for+low%2Dresource+language+understanding%2E+arXiv+preprint)