---
title: "Investigating the Translation Performance of a Large Multilingual"
description: "The authors release their code and outputs, which supports reproducible evaluation of… 19.3 BLEU. 54 references, cited by 74 papers here."
authors: ["Rachel Bawden", "Fran\\ccois Yvon"]
year: 2023
venue: "arXiv:2303.01911"
source: "https://arxiv.org/abs/2303.01911"
doi: "10.48550/arxiv.2303.01911"
arxiv: "2303.01911"
licence: "CC BY-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bawden-2023-investigating-translation-performance-large-multilingual/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bawden-2023-investigating-translation-performance-large-multilingual/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM

# Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM

**Authors:** [Rachel Bawden](https://md.research.elamur.ai/free/en/authors/rachel-bawden/) and Fran\\ccois Yvon · **Year:** 2023 · **Venue:** arXiv:2303.01911 · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2303.01911) · [PDF](https://arxiv.org/pdf/2303.01911) · [DOI 10.48550/arxiv.2303.01911](https://doi.org/10.48550/arxiv.2303.01911) · [arXiv 2303.01911](https://arxiv.org/abs/2303.01911) · [OpenAlex](https://openalex.org/W4323323154)

**Citations:** 11 citations (OpenAlex, 2026-10-14) · [74](https://md.research.elamur.ai/free/en/papers/bawden-2023-investigating-translation-performance-large-multilingual/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Large language models are often tested on many tasks at once, so their translation ability, especially beyond a few well-resourced language pairs, is poorly understood. Bawden and Yvon examine the open-access multilingual model BLOOM, which covers 46 languages, as a machine translation system. They test it on three benchmarks: WMT 2014 news (English↔French and English↔Hindi), the multi-parallel Flores-101 for high- and low-resource pairs, and DiaBLa, a set of informal English–French dialogues. They use the Language Model Evaluation Harness with several prompt templates, 0-shot and few-shot settings, and different model sizes, and they score outputs with BLEU and COMET.

Without examples, BLOOM tends to keep generating past the translation and sometimes answers in the wrong language. Adding even a few examples greatly reduces both problems and brings many pairs close to state-of-the-art results. Some low-resource pairs remain weak. The study also finds cross-lingual transfer and shows that preceding dialogue context affects the translations, even though scores do not rise. The authors release their code and outputs, which supports reproducible evaluation of open LLMs for MT.

## Key points
- BLOOM was trained on about 1.6TB of text and released in sizes from 560M to 176B parameters. English makes up roughly 30% of the data, French 12.9% and Hindi 0.7%. Part of the data comes from the ROOTS corpus.
- The comparison systems were T0, mT0-XXL and OPT, all run with the xglm-source+target prompt. The supervised 615M-parameter M2M-100 served as the Flores-101 topline.
- Few-shot examples from a related, higher-resource direction (hi→en) gave the best transfer results, while the unrelated fr→ar direction gave the worst. In-domain Flores-101 examples beat WMT ones.
- On DiaBLa, the direction of the 1-shot example matters a lot for overgeneration. For en→fr before truncation, a same-direction example scored 19.3 BLEU and an opposite-direction one scored 3.6.
- Using the previous utterance instead of a random example as context gave mixed metric results. However, it visibly changed the outputs, for example in word-sense choice and pronoun gender agreement.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Institut national de recherche en sciences et technologies du numérique](https://md.research.elamur.ai/free/en/labs/institut-national-de-recherche-en-sciences-et-technologies-du-numerique/) — Research institute, Le Chesnay-Rocquencourt, France · 62 works · 17,271 citations
- [Université Paris-Saclay](https://md.research.elamur.ai/free/en/universities/universite-paris-saclay/) — University, Gif-sur-Yvette, France · 34 works · 5,733 citations
- [ALMANACH: Modélisation et analyse linguistique automatique et humanités computationnelles](https://md.research.elamur.ai/free/en/labs/almanach-modelisation-et-analyse-linguistique-automatique-et-humanites/) — Research institute, Paris, France · 15 works · 6,780 citations
- [Laboratoire Interdisciplinaire des Sciences du Numérique](https://md.research.elamur.ai/free/en/labs/laboratoire-interdisciplinaire-des-sciences-du-numerique/) — Research institute, Gif-sur-Yvette, France · 10 works · 422 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 467 works

## Cite

Cite as: Rachel Bawden and Fran\\ccois Yvon. 2023. Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM. arXiv:2303.01911.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{bawden2023investigating,
  title = {{Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM}},
  author = {Rachel Bawden and Franccois Yvon},
  year = {2023},
  doi = {10.48550/arxiv.2303.01911},
  eprint = {2303.01911},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.01911}
}
```

- [← previous InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition](https://md.research.elamur.ai/free/en/papers/zhang-2023-internlm-xcomposer-vision-language/)
- [next → Multilingual Large Language Models Are Not (Yet) Code-Switchers](https://md.research.elamur.ai/free/en/papers/zhang-2023-multilingual-large-language-models-are/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

54 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya… [find](https://scholar.google.com/scholar?q=Stephen+Bach%2C+Victor+Sanh%2C+Zheng+Xin+Yong%2C+Albert+Webson%2C+Colin+Raffel%2C+Nihal+V%2E+Nayak%2C+Abheesht+Sharma%2C+Taewoon+Kim%2C+M+Saiful+Bari%2C+Thibault+Fevry%2C+Zaid+Alyafeai%2C+Manan+Dey%2C+Andrea+Santilli%2C+Zhiqing+)
2. Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Firat, Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey, Theresa Breiner, Vera Axelrod, Jason Riesa, Yuan Cao, Mia Xu Chen, Klaus Macherey, Maxim Krikun, Pidong Wang, Alexander Gutkin, Apurva Shah, Yanp… — [arXiv:2205.03983](https://arxiv.org/abs/2205.03983) [find](https://scholar.google.com/scholar?q=Ankur+Bapna%2C+Isaac+Caswell%2C+Julia+Kreutzer%2C+Orhan+Firat%2C+Daan+van+Esch%2C+Aditya+Siddhant%2C+Mengmeng+Niu%2C+Pallavi+Baljekar%2C+Xavier+Garcia%2C+Wolfgang+Macherey%2C+Theresa+Breiner%2C+Vera+Axelrod%2C+Jason+Riesa%2C+Y)
3. Rachel Bawden, Eric Bilinski, Thomas Lavergne, and Sophie Rosset. 2021. DiaBLa: a corpus of bilingual spontaneous written dialogues for machine translation. Language Resources and Evaluation, 55(3):635–660. [find](https://scholar.google.com/scholar?q=Rachel+Bawden%2C+Eric+Bilinski%2C+Thomas+Lavergne%2C+and+Sophie+Rosset%2E+2021%2E+DiaBLa%3A+a+corpus+of+bilingual+spontaneous+written+dialogues+for+machine+translation%2E+Language+Resources+and+Evaluation%2C+55%283%29%3A63)
4. Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. [Evaluating discourse phenomena in neural machine translation](https://md.research.elamur.ai/free/en/papers/bawden-2017-evaluating-discourse-phenomena-neural-machine/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Paper…
5. Bin Bi, Chenliang Li, Chen Wu, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si. 2020. PALM: Pre-training an autoencoding&autoregressive language model for context-conditioned generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), page… [find](https://scholar.google.com/scholar?q=Bin+Bi%2C+Chenliang+Li%2C+Chen+Wu%2C+Ming+Yan%2C+Wei+Wang%2C+Songfang+Huang%2C+Fei+Huang%2C+and+Luo+Si%2E+2020%2E+PALM%3A+Pre%2Dtraining+an+autoencoding%26autoregressive+language+model+for+context%2Dconditioned+generation%2E+In+)
6. Workshop BigScience, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoı̂… [find](https://scholar.google.com/scholar?q=Workshop+BigScience%2C+Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagne%CC%81%2C+Alexandra+Sasha+Luccioni%2C+Franc%CC%A7ois+Yvon%2C+Matthias+Galle%CC%81%2C+Jonathan+Tow)
7. Terra Blevins and Luke Zettlemoyer. 2022. Language contamination helps explains the cross-lingual capabilities of English pretrained models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3563–3574, Abu Dhabi, United Arab Emirates. Association for Co… [find](https://scholar.google.com/scholar?q=Terra+Blevins+and+Luke+Zettlemoyer%2E+2022%2E+Language+contamination+helps+explains+the+cross%2Dlingual+capabilities+of+English+pretrained+models%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+)
8. Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. [Findings of the 2014 workshop on statistical machine translation](https://md.research.elamur.ai/free/en/papers/bojar-2014-findings-2014-workshop-statistical-machine/). In Proceedings of the…
9. Ondřej Bojar and Aleš Tamchyna. 2011. Improving translation model by monolingual data. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 330–336, Edinburgh, Scotland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar+and+Ales%CC%8C+Tamchyna%2E+2011%2E+Improving+translation+model+by+monolingual+data%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2C+pages+330%E2%80%93336%2C+Edinburgh%2C+Scotland%2E+As)
10. [Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
11. [Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi…](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
12. Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loı̈c Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Se… [find](https://scholar.google.com/scholar?q=Marta+R%2E+Costa%2Djussa%CC%80%2C+James+Cross%2C+Onur+C%CC%A7elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+Jean+Maillard%2C+Anna+Sun%2C+Skyler+Wang%2C+Guillaume+Wenzek%2C+Al+)
13. Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. [Multi-task learning for multiple language translation](https://md.research.elamur.ai/free/en/papers/dong-2015-multi-task-learning-multiple-language/). In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Lon…
14. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Armand Joulin. 2021. Beyond English-Centric multilingual machine tra… [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Shruti+Bhosale%2C+Holger+Schwenk%2C+Zhiyi+Ma%2C+Ahmed+El%2DKishky%2C+Siddharth+Goyal%2C+Mandeep+Baines%2C+Onur+Celebi%2C+Guillaume+Wenzek%2C+Vishrav+Chaudhary%2C+Naman+Goyal%2C+Tom+Birch%2C+Vitaliy+Liptchinsky%2C+S)
15. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. [Multi-way, multilingual neural machine translation with a shared attention mechanism](https://md.research.elamur.ai/free/en/papers/firat-2016-multi-way-multilingual-neural-machine/). In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 866–875,…
16. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Jason+Phang%2C+Laria+Reynolds%2C+Eric+Tang%2C+Anish+Thite)
17. Xavier Garcia, Yamini Bansal, Colin Cherry, George F. Foster, Maxim Krikun, Fangxiaoyu Feng, Melvin Johnson, and Orhan Firat. 2023. The unreasonable effectiveness of few-shot learning for machine translation. CoRR, abs/2302.01398. — [arXiv:2302.01398](https://arxiv.org/abs/2302.01398) [find](https://scholar.google.com/scholar?q=Xavier+Garcia%2C+Yamini+Bansal%2C+Colin+Cherry%2C+George+F%2E+Foster%2C+Maxim+Krikun%2C+Fangxiaoyu+Feng%2C+Melvin+Johnson%2C+and+Orhan+Firat%2E+2023%2E+The+unreasonable+effectiveness+of+few%2Dshot+learning+for+machine+tran)
18. Xavier Garcia and Orhan Firat. 2022. Using natural language prompts for machine translation. CoRR, abs/2202.11822. — [arXiv:2202.11822](https://arxiv.org/abs/2202.11822) [find](https://scholar.google.com/scholar?q=Xavier+Garcia+and+Orhan+Firat%2E+2022%2E+Using+natural+language+prompts+for+machine+translation%2E+CoRR%2C+abs%2F2202%2E11822%2E+%E2%80%94+%5BarXiv%3A2202%2E11822%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2202%2E11822%29)
19. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computa… [find](https://scholar.google.com/scholar?q=Naman+Goyal%2C+Cynthia+Gao%2C+Vishrav+Chaudhary%2C+Peng%2DJen+Chen%2C+Guillaume+Wenzek%2C+Da+Ju%2C+Sanjana+Krishnan%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Francisco+Guzma%CC%81n%2C+and+Angela+Fan%2E+2022%2E+The+Flores%2D101+evaluation+benchmar)
20. Christian Hardmeier. 2012. Discourse in statistical machine translation: A survey and a case study. Discours - Revue de linguistique, psycholinguistique et informatique, (11). [find](https://scholar.google.com/scholar?q=Christian+Hardmeier%2E+2012%2E+Discourse+in+statistical+machine+translation%3A+A+survey+and+a+case+study%2E+Discours+%2D+Revue+de+linguistique%2C+psycholinguistique+et+informatique%2C+%2811%29%2E)
21. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are GPT models at machine translation? a comprehensive evaluation. CoRR, abs/2302.09210. [find](https://scholar.google.com/scholar?q=Amr+Hendy%2C+Mohamed+Abdelrehim%2C+Amr+Sharaf%2C+Vikas+Raunak%2C+Mohamed+Gabr%2C+Hitokazu+Matsushita%2C+Young+Jin+Kim%2C+Mohamed+Afify%2C+and+Hany+Hassan+Awadalla%2E+2023%2E+How+good+are+GPT+models+at+machine+translation)
22. Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. [Is ChatGPT a Good Translator?](https://md.research.elamur.ai/free/en/papers/jiao-2023-chatgpt-good-translator-yes-gpt/) A Preliminary Study. CoRR, abs/2301.08745.
23. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, Valencia, Spain. Associatio… [find](https://scholar.google.com/scholar?q=Armand+Joulin%2C+Edouard+Grave%2C+Piotr+Bojanowski%2C+and+Tomas+Mikolov%2E+2017%2E+Bag+of+tricks+for+efficient+text+classification%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Associati)
24. [Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gérard Dupont, Stella Biderman, Anna Rogers…](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2023-bigscience-roots-corpus-1-6tb/)
25. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Brian+Lester%2C+Rami+Al%2DRfou%2C+and+Noah+Constant%2E+2021%2E+The+power+of+scale+for+parameter%2Defficient+prompt+tuning%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processing)
26. Jindřich Libovický and Jindřich Helcl. 2017. [Attention strategies for multi-source sequence-to-sequence learning](https://md.research.elamur.ai/free/en/papers/libovicky-2017-attention-strategies-multi-source-sequence/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 196–202, Vancouver, Canada. Association for Computational Li…
27. [Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyano…](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
28. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. [Pre– Train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/). ACM Computing Surveys, 55(9).
29. [Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Tr…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
30. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
31. António Lopes, M. Amin Farajian, Rachel Bawden, Michael Zhang, and André F. T. Martins. 2020. [Document-level neural MT: A systematic comparison](https://md.research.elamur.ai/free/en/papers/lopes-2020-document-level-neural-mt-systematic/). In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 225–234, Lisboa, Portugal. European Association f…
32. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. [Language models as knowledge bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural…
33. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. [Multi-task sequence to sequence learning](https://md.research.elamur.ai/free/en/papers/luong-2015-multi-task-sequence-sequence-learning/). In Proceedings of the 4th International Conference on Learning Representations, San Juan, Puerto Rico.
34. Minh Quang Pham, Jitao Xu, Josep Crego, François Yvon, and Jean Senellart. 2020. Priming neural machine translation. In Proceedings of the Fifth Conference on Machine Translation, pages 516–527, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Minh+Quang+Pham%2C+Jitao+Xu%2C+Josep+Crego%2C+Franc%CC%A7ois+Yvon%2C+and+Jean+Senellart%2E+2020%2E+Priming+neural+machine+translation%2E+In+Proceedings+of+the+Fifth+Conference+on+Machine+Translation%2C+pages+516%E2%80%93527%2C+Onli)
35. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. [The natural language decathlon: Multitask learning as question answering](https://md.research.elamur.ai/free/en/papers/mccann-2018-natural-language-decathlon-multitask-learning/). CoRR, abs/1806.08730.
36. Matt Post. 2018. [A call for clarity in reporting BLEU scores](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/). In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics.
37. Yasmin Moslem, Rejwanul Haque, and Andy Way. 2023. [Adaptive machine translation with large language models](https://md.research.elamur.ai/free/en/papers/moslem-2023-adaptive-machine-translation-large-language/). CoRR, abs/2301.13294.
38. [Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel…](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/)
39. Prashanth Nayak, Rejwanul Haque, John D Kelleher, and Andy Way. 2022. Investigating Contextual Influence in Document-Level Translation. Information, 13(5):249. [find](https://scholar.google.com/scholar?q=Prashanth+Nayak%2C+Rejwanul+Haque%2C+John+D+Kelleher%2C+and+Andy+Way%2E+2022%2E+Investigating+Contextual+Influence+in+Document%2DLevel+Translation%2E+Information%2C+13%285%29%3A249%2E)
40. Pedro Javier Ortiz Suárez, Benoı̂t Sagot, and Laurent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. In Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7), pages 9–16, Mannheim. Leibniz-Institut für Deu… [find](https://scholar.google.com/scholar?q=Pedro+Javier+Ortiz+Sua%CC%81rez%2C+Beno%C4%B1%CC%82t+Sagot%2C+and+Laurent+Romary%2E+2019%2E+Asynchronous+pipelines+for+processing+huge+corpora+on+medium+to+low+resource+infrastructures%2E+In+Proceedings+of+the+Workshop+on+Cha)
41. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+Technical+report%2E)
42. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to– text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67.
43. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. [COMET: A neural framework for MT evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.
44. [Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chan…](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
45. Timo Schick and Hinrich Schütze. 2021. [It’s not just size that matters: Small language models are also few-shot learners](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339–2352, Online. Assoc…
46. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235, Onlin…
47. Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith Peris, Stephen Rawls, Andy Rosenbaum, Anna Rumshisky, Chandana Satya Prakash, Mukund Sridhar, Fabian Triefenbach, Apurv Verma, Gökhan Tür, and Prem Natarajan. 2022. Alexatm 20b: Few-shot learning… [find](https://scholar.google.com/scholar?q=Saleh+Soltan%2C+Shankar+Ananthakrishnan%2C+Jack+FitzGerald%2C+Rahul+Gupta%2C+Wael+Hamza%2C+Haidar+Khan%2C+Charith+Peris%2C+Stephen+Rawls%2C+Andy+Rosenbaum%2C+Anna+Rumshisky%2C+Chandana+Satya+Prakash%2C+Mukund+Sridhar%2C+Fabi)
48. Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. [Attaining the unattainable? reassessing claims of human parity in neural machine translation](https://md.research.elamur.ai/free/en/papers/toral-2018-attaining-unattainable-reassessing-claims-human/). In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 113–123, Brussels, Belgium. Association for Computational Lin…
49. David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George F. Foster. 2022. Prompting PaLM for Translation: Assessing Strategies and Performance. CoRR, abs/2211.09102. [find](https://scholar.google.com/scholar?q=David+Vilar%2C+Markus+Freitag%2C+Colin+Cherry%2C+Jiaming+Luo%2C+Viresh+Ratnakar%2C+and+George+F%2E+Foster%2E+2022%2E+Prompting+PaLM+for+Translation%3A+Assessing+Strategies+and+Performance%2E+CoRR%2C+abs%2F2211%2E09102%2E)
50. Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. [Context-aware neural machine translation learns anaphora resolution](https://md.research.elamur.ai/free/en/papers/voita-2018-context-aware-neural-machine-translation/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1264–1274, Melbourne, Australia. Associati…
51. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Proceedings of the 36th Annual Conference on Neural Information Processing Systems, New Orleans, Louisiana,…
52. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, and Jie Tang. 2022. [GLM-130B: an open bilingual pre-trained model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/). CoRR, abs/2210.02414.
53. Biao Zhang, Barry Haddow, and Alexandra Birch. 2023. [Prompting large language model for machine translation: A case study](https://md.research.elamur.ai/free/en/papers/zhang-2023-prompting-large-language-model-machine/). CoRR, abs/2301.07069.
54. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. [OPT: Open pre-trai…](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/)

*Updated 24 October 2026*