---
title: "ChatGPT MT: Competitive for High — Robinson 2023"
description: "LLM chatbots are now widely used by ordinary people for translation, yet for most of the world's languages there was no published evidence on how well…"
authors: ["Nathaniel Robinson", "Perez Ogayo", "David R. Mortensen", "Graham Neubig"]
year: 2023
venue: "Proceedings of the Eighth Conference on Machine Translation"
source: "https://aclanthology.org/2023.wmt-1.40/"
doi: "10.18653/v1/2023.wmt-1.40"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/robinson-2023-chatgpt-mt-competitive-high-but/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/robinson-2023-chatgpt-mt-competitive-high-but/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2023](https://md.research.elamur.ai/free/papers/2023/page/2/) / ChatGPT MT: Competitive for High- (but Not Low-) Resource Languages

# ChatGPT MT: Competitive for High- (but Not Low-) Resource Languages

**Authors:** Nathaniel Robinson, Perez Ogayo, [David R. Mortensen](https://md.research.elamur.ai/free/authors/david-r-mortensen/), [Graham Neubig](https://md.research.elamur.ai/free/authors/graham-neubig/) · **Year:** 2023 · **Venue:** Proceedings of the Eighth Conference on Machine Translation · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.wmt-1.40/) · [PDF](https://aclanthology.org/2023.wmt-1.40.pdf) · [DOI 10.18653/v1/2023.wmt-1.40](https://doi.org/10.18653/v1/2023.wmt-1.40) · [OpenAlex](https://openalex.org/W4389524378)

**Citations:** 51 citations (openalex-snapshot · 2026-10-14) · [100](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
LLM chatbots are now widely used by ordinary people for translation, yet for most of the world's languages there was no published evidence on how well recent models translate them. The authors measured English-to-X translation by ChatGPT (gpt-3.5-turbo) on the FLORES-200 benchmark, which covers 204 language varieties. They avoided the reverse direction because the English side comes from Wikipedia and likely appeared in OpenAI's training data. ChatGPT was tested with zero- and five-shot prompts adapted from Gao et al. (2023) and compared against NLLB-MOE, Google Translate, and GPT-4 on a subset of languages. Scoring used chrF++ and spBLEU. Learned metrics such as COMET were excluded because they cover few low-resource languages. The authors also analysed which language features predict ChatGPT's relative quality and compared costs across systems. The LLMs come close to dedicated MT systems for some well-resourced languages but fall clearly behind for low-resource ones, especially African languages. Resource level turned out to be the strongest predictor. These findings help speakers decide when an LLM is a reasonable translation tool.

## Key points
- **Average chrF by system:** Google Translate 52.2 (115 languages), NLLB 45.3 (201), GPT-4 44.6 (20), ChatGPT five-shot 33.1 and zero-shot 32.3 (203).
- **ChatGPT vs. baselines:** traditional systems beat ChatGPT on 84.1% of the evaluated languages. Five-shot prompting adds less than one chrF point over zero-shot on average.
- **GPT-4 vs. ChatGPT:** GPT-4 was tested only on 20 languages because it costs about 1900% more than ChatGPT. It improved on ChatGPT by 6.5 chrF on average, with the largest gains for low-resource languages, and outperformed NLLB on five of them.
- **Ukrainian (ukr_Cyrl):** chrF was 56.3 for GPT-4, 55.4 for ChatGPT, and 56.3 for NLLB, so the LLMs are roughly on par with dedicated MT here.
- **Resources:** a decision-tree analysis identified resource level (proxied by Wikipedia page counts) as the main factor behind ChatGPT's relative performance. Code and outputs are released in the cmu-llab/gpt_mt_benchmark repository, with a Zeno browser for exploring scores.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/)

## Cited by

Cited by

100 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 91 not yet.

- [Many-Shot In-Context Learning](https://md.research.elamur.ai/free/papers/agarwal-2024-many-shot-context-learning/)
- [LLM-Assisted Rule Based Machine Translation for Low/No-Resource Languages](https://md.research.elamur.ai/free/papers/coleman-2024-llm-assisted-rule-based-machine/)
- [Context Volume Drives Performance: Tackling Domain Shift in Extremely Low-Resource Translation via RAG](https://md.research.elamur.ai/free/papers/lingbulk-2026-setiawan-context-volume-drives-performance-tackling/)
- [New Trends for Modern Machine Translation with Large Reasoning Models](https://md.research.elamur.ai/free/papers/liu-2026-new-trends-modern-machine-translation/)
- [GrammaMT: Improving Machine Translation with Grammar-Informed In-Context Learning](https://md.research.elamur.ai/free/papers/ramos-2025-grammamt-improving-machine-translation-grammar/)
- [AMIYA Shared Task: Arabic Modeling In Your Accent at VarDial 2026](https://md.research.elamur.ai/free/papers/robinson-2026-amiya-shared-task-arabic-modeling/)
- [Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation](https://md.research.elamur.ai/free/papers/singh-2025-global-mmlu-understanding-addressing-cultural/)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://md.research.elamur.ai/free/papers/ustun-2024-aya-model-instruction-finetuned-open/)
- [Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis](https://md.research.elamur.ai/free/papers/zhu-2024-multilingual-machine-translation-large-language/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{robinson2023chatgpt,
  title = {{ChatGPT MT: Competitive for High- (but Not Low-) Resource Languages}},
  author = {Robinson, Nathaniel and Ogayo, Perez and Mortensen, David R. and Neubig, Graham},
  booktitle = {Proceedings of the Eighth Conference on Machine Translation},
  year = {2023},
  doi = {10.18653/v1/2023.wmt-1.40},
  url = {https://doi.org/10.18653/v1/2023.wmt-1.40}
}
```

- [← previous ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education](https://md.research.elamur.ai/free/papers/kasneci-2023-chatgpt-good-opportunities-challenges-large/)
- [next → ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://md.research.elamur.ai/free/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/papers/2023/page/2/)

## References

29 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Radford%2C+Ilya+Sutskever%2C+and+Dario+Amodei%2E+2020%2E+Language+models+are+few%2Dshot+learners%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+33%2C+pages+1877%E2%80%931901%2E+Curran+Associates%2C+Inc%2E)
2. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2020. Beyond english-centric multiling… [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Shruti+Bhosale%2C+Holger+Schwenk%2C+Zhiyi+Ma%2C+Ahmed+El%2DKishky%2C+Siddharth+Goyal%2C+Mandeep+Baines%2C+Onur+Celebi%2C+Guillaume+Wenzek%2C+Vishrav+Chaudhary%2C+Naman+Goyal%2C+Tom+Birch%2C+Vitaliy+Liptchinsky%2C+S)
3. Yuan Gao, Ruili Wang, and Feng Hou. 2023. How to design translation prompts for chatgpt: An empirical study. arXiv e-prints, pages arXiv–2304. [find](https://scholar.google.com/scholar?q=Yuan+Gao%2C+Ruili+Wang%2C+and+Feng+Hou%2E+2023%2E+How+to+design+translation+prompts+for+chatgpt%3A+An+empirical+study%2E+arXiv+e%2Dprints%2C+pages+arXiv%E2%80%932304%2E)
4. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computat… [find](https://scholar.google.com/scholar?q=Naman+Goyal%2C+Cynthia+Gao%2C+Vishrav+Chaudhary%2C+Peng%2DJen+Chen%2C+Guillaume+Wenzek%2C+Da+Ju%2C+Sanjana+Krishnan%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Francisco+Guzm%C3%A1n%2C+and+Angela+Fan%2E+2022%2E+The+Flores%2D101+evaluation+benchmark)
5. Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022. Towards a Cleaner Document-Oriented Multilingual Crawled Corpus. arXiv eprints, page arXiv:2201.06642. [find](https://scholar.google.com/scholar?q=Julien+Abadji%2C+Pedro+Ortiz+Suarez%2C+Laurent+Romary%2C+and+Beno%C3%AEt+Sagot%2E+2022%2E+Towards+a+Cleaner+Document%2DOriented+Multilingual+Crawled+Corpus%2E+arXiv+eprints%2C+page+arXiv%3A2201%2E06642%2E) [our card](https://md.research.elamur.ai/free/papers/abadji-2022-towards-cleaner-document-oriented-multilingual/)
6. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at machine translation? a comprehensive evaluation. arXiv preprint arXiv:2302.09210. [find](https://scholar.google.com/scholar?q=Amr+Hendy%2C+Mohamed+Abdelrehim%2C+Amr+Sharaf%2C+Vikas+Raunak%2C+Mohamed+Gabr%2C+Hitokazu+Matsushita%2C+Young+Jin+Kim%2C+Mohamed+Afify%2C+and+Hany+Hassan+Awadalla%2E+2023%2E+How+good+are+gpt+models+at+machine+translation)
7. Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, and Yulia Tsvetkov. 2023. Do all languages cost the same? tokenization in the era of commercial language models. [find](https://scholar.google.com/scholar?q=Orevaoghene+Ahia%2C+Sachin+Kumar%2C+Hila+Gonen%2C+Jungo+Kasai%2C+David+R%2E+Mortensen%2C+Noah+A%2E+Smith%2C+and+Yulia+Tsvetkov%2E+2023%2E+Do+all+languages+cost+the+same%3F+tokenization+in+the+era+of+commercial+language+mod) [our card](https://md.research.elamur.ai/free/papers/ahia-2023-do-all-languages-cost-same/)
8. Wenxiang Jiao, WX Wang, JT Huang, Xing Wang, and ZP Tu. 2023. Is chatgpt a good translator? yes with gpt-4 as the engine. arXiv preprint arXiv:2301.08745. [find](https://scholar.google.com/scholar?q=Wenxiang+Jiao%2C+WX+Wang%2C+JT+Huang%2C+Xing+Wang%2C+and+ZP+Tu%2E+2023%2E+Is+chatgpt+a+good+translator%3F+yes+with+gpt%2D4+as+the+engine%2E+arXiv+preprint+arXiv%3A2301%2E08745%2E) [our card](https://md.research.elamur.ai/free/papers/jiao-2023-chatgpt-good-translator-yes-gpt/)
9. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. [find](https://scholar.google.com/scholar?q=Yejin+Bang%2C+Samuel+Cahyawijaya%2C+Nayeon+Lee%2C+Wenliang+Dai%2C+Dan+Su%2C+Bryan+Wilie%2C+Holy+Lovenia%2C+Ziwei+Ji%2C+Tiezheng+Yu%2C+Willy+Chung%2C+Quyet+V%2E+Do%2C+Yan+Xu%2C+and+Pascale+Fung%2E+2023%2E+A+multitask%2C+multilingual%2C) [our card](https://md.research.elamur.ai/free/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/)
10. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Jiatao+Gu%2C+Naman+Goyal%2C+Xian+Li%2C+Sergey+Edunov%2C+Marjan+Ghazvininejad%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2020%2E+Multilingual+denoising+pre%2Dtraining+for+neural+machine+translation%2E+Transactio)
11. Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Firat, Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey, Theresa Breiner, Vera Axelrod, Jason Riesa, Yuan Cao, Mia Xu Chen, Klaus Macherey, Maxim Krikun, Pidong Wang, Alexander Gutkin, Apurva Shah, Yanp… [find](https://scholar.google.com/scholar?q=Ankur+Bapna%2C+Isaac+Caswell%2C+Julia+Kreutzer%2C+Orhan+Firat%2C+Daan+van+Esch%2C+Aditya+Siddhant%2C+Mengmeng+Niu%2C+Pallavi+Baljekar%2C+Xavier+Garcia%2C+Wolfgang+Macherey%2C+Theresa+Breiner%2C+Vera+Axelrod%2C+Jason+Riesa%2C+Y)
12. Yasmin Moslem, Rejwanul Haque, and Andy Way. 2023. Adaptive machine translation with large language models. arXiv preprint arXiv:2301.13294. [find](https://scholar.google.com/scholar?q=Yasmin+Moslem%2C+Rejwanul+Haque%2C+and+Andy+Way%2E+2023%2E+Adaptive+machine+translation+with+large+language+models%2E+arXiv+preprint+arXiv%3A2301%2E13294%2E) [our card](https://md.research.elamur.ai/free/papers/moslem-2023-adaptive-machine-translation-large-language/)
13. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretchen)
14. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia-Gonzalez, Prangthip Hansanti, John Hof… [find](https://scholar.google.com/scholar?q=NLLB+Team%2C+Marta+R%2E+Costa%2Djuss%C3%A0%2C+James+Cross%2C+Onur+%C3%87elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+Jean+Maillard%2C+Anna+Sun%2C+Skyler+Wang%2C+Guillaume+We)
15. OpenAI. 2023. Gpt-4 technical report. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023%2E+Gpt%2D4+technical+report%2E) [our card](https://md.research.elamur.ai/free/papers/openai-2023-gpt-4-technical-report/)
16. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, page 311–318, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+A+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+on+Association+for+Computat) [our card](https://md.research.elamur.ai/free/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
17. Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maja+Popovic%CC%81%2E+2017%2E+chrF%2B%2B%3A+words+helping+character+n%2Dgrams%2E+In+Proceedings+of+the+Second+Conference+on+Machine+Translation%2C+pages+612%E2%80%93618%2C+Copenhagen%2C+Denmark%2E+Association+for+Computational+Linguist) [our card](https://md.research.elamur.ai/free/papers/popovic-2017-chrf-words-helping-character-n/)
18. Liang Xu, Anqi Li, Lei Zhu, Hang Xue, Changtai Zhu, Kangkang Zhao, Haonan He, Xuanwei Zhang, Qiyue Kang, and Zhenzhong Lan. 2023. Superclue: A comprehensive chinese large language model benchmark. [find](https://scholar.google.com/scholar?q=Liang+Xu%2C+Anqi+Li%2C+Lei+Zhu%2C+Hang+Xue%2C+Changtai+Zhu%2C+Kangkang+Zhao%2C+Haonan+He%2C+Xuanwei+Zhang%2C+Qiyue+Kang%2C+and+Zhenzhong+Lan%2E+2023%2E+Superclue%3A+A+comprehensive+chinese+large+language+model+benchmark%2E)
19. Wenxuan Zhang, Sharifah Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. [find](https://scholar.google.com/scholar?q=Wenxuan+Zhang%2C+Sharifah+Mahani+Aljunied%2C+Chang+Gao%2C+Yew+Ken+Chia%2C+and+Lidong+Bing%2E+2023%2E+M3exam%3A+A+multilingual%2C+multimodal%2C+multilevel+benchmark+for+examining+large+language+models%2E)
20. Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. Multilingual machine translation with large language models: Empirical results and analysis. [find](https://scholar.google.com/scholar?q=Wenhao+Zhu%2C+Hongyi+Liu%2C+Qingxiu+Dong%2C+Jingjing+Xu%2C+Shujian+Huang%2C+Lingpeng+Kong%2C+Jiajun+Chen%2C+and+Lei+Li%2E+2023%2E+Multilingual+machine+translation+with+large+language+models%3A+Empirical+results+and+analy) [our card](https://md.research.elamur.ai/free/papers/zhu-2024-multilingual-machine-translation-large-language/)
21. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Belgium, Brussels. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Matt+Post%2E+2018%2E+A+call+for+clarity+in+reporting+BLEU+scores%2E+In+Proceedings+of+the+Third+Conference+on+Machine+Translation%3A+Research+Papers%2C+pages+186%E2%80%93+191%2C+Belgium%2C+Brussels%2E+Association+for+Computa) [our card](https://md.research.elamur.ai/free/papers/post-2018-call-clarity-reporting-bleu-scores/)
22. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
23. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Comet: A neural framework for mt evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Craig+Stewart%2C+Ana+C+Farinha%2C+and+Alon+Lavie%2E+2020%2E+Comet%3A+A+neural+framework+for+mt+evaluation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+) [our card](https://md.research.elamur.ai/free/papers/rei-2020-comet-neural-framework-mt-evaluation/)
24. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+et+al%2E+2022%2E+Bloom%3A+A+176b%2Dparameter)
25. Michael Schade. 2023. How your data is used to improve model performance. [find](https://scholar.google.com/scholar?q=Michael+Schade%2E+2023%2E+How+your+data+is+used+to+improve+model+performance%2E)
26. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. Bleurt: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881– 7892. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Dipanjan+Das%2C+and+Ankur+Parikh%2E+2020%2E+Bleurt%3A+Learning+robust+metrics+for+text+generation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+)
27. Aditya Siddhant, Ankur Bapna, Orhan Firat, Yuan Cao, Mia Xu Chen, Isaac Caswell, and Xavier Garcia. 2022. Towards the next 1000 languages in multilingual machine translation: Exploring the synergy between supervised and self-supervised learning. CoRR, abs/2201.03110. — [arXiv:2201.03110](https://arxiv.org/abs/2201.03110) [find](https://scholar.google.com/scholar?q=Aditya+Siddhant%2C+Ankur+Bapna%2C+Orhan+Firat%2C+Yuan+Cao%2C+Mia+Xu+Chen%2C+Isaac+Caswell%2C+and+Xavier+Garcia%2E+2022%2E+Towards+the+next+1000+languages+in+multilingual+machine+translation%3A+Exploring+the+synergy+bet)
28. MosaicML NLP Team. 2023. Introducing mpt-7b: A new standard for open-source, commercially usable llms. Accessed: 2023-05-05. [find](https://scholar.google.com/scholar?q=MosaicML+NLP+Team%2E+2023%2E+Introducing+mpt%2D7b%3A+A+new+standard+for+open%2Dsource%2C+commercially+usable+llms%2E+Accessed%3A+2023%2D05%2D05%2E)
29. Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. 2023. Document-level machine translation with large language models. arXiv preprint arXiv:2304.02210. [find](https://scholar.google.com/scholar?q=Longyue+Wang%2C+Chenyang+Lyu%2C+Tianbo+Ji%2C+Zhirui+Zhang%2C+Dian+Yu%2C+Shuming+Shi%2C+and+Zhaopeng+Tu%2E+2023%2E+Document%2Dlevel+machine+translation+with+large+language+models%2E+arXiv+preprint+arXiv%3A2304%2E02210%2E) [our card](https://md.research.elamur.ai/free/papers/wang-2023-document-level-machine-translation-large/)