---
title: "WMT24++ — paper by Deutsch et al. (2025) — summary and ideas"
description: "This paper builds WMT24++, which reuses the 998 paragraph-level English sources of WMT24… 42 references, cited by 74 papers here."
authors: ["Daniel Deutsch", "Eleftheria Briakou", "Isaac Rayburn Caswell", "Mara Finkelstein", "Rebecca Galor", "Juraj Juraska", "Geza Kovacs", "Alison Lui", "Ricardo Rei", "Jason Riesa", "Shruti Rijhwani", "Parker Riley", "Elizabeth Salesky", "Firas Trabelsi", "Stephanie Winkler", "Biao Zhang", "Markus Freitag"]
year: 2025
venue: "Findings of the Association for Computational Linguistics: ACL 2025"
source: "https://aclanthology.org/2025.findings-acl.634/"
doi: "10.18653/v1/2025.findings-acl.634"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/deutsch-2025-wmt24-expanding-language-coverage-wmt24/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/deutsch-2025-wmt24-expanding-language-coverage-wmt24/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/) / WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects

# WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects

**Authors:** [Daniel Deutsch](https://md.research.elamur.ai/free/en/authors/daniel-deutsch/), [Eleftheria Briakou](https://md.research.elamur.ai/free/en/authors/eleftheria-briakou/), [Isaac Rayburn Caswell](https://md.research.elamur.ai/free/en/authors/isaac-caswell/), [Mara Finkelstein](https://md.research.elamur.ai/free/en/authors/mara-finkelstein/), Rebecca Galor, [Juraj Juraska](https://md.research.elamur.ai/free/en/authors/juraj-juraska/), Geza Kovacs, Alison Lui, [Ricardo Rei](https://md.research.elamur.ai/free/en/authors/ricardo-rei/), [Jason Riesa](https://md.research.elamur.ai/free/en/authors/jason-riesa/), [Shruti Rijhwani](https://md.research.elamur.ai/free/en/authors/shruti-rijhwani/), [Parker Riley](https://md.research.elamur.ai/free/en/authors/parker-riley/), [Elizabeth Salesky](https://md.research.elamur.ai/free/en/authors/elizabeth-salesky/), Firas Trabelsi, Stephanie Winkler, [Biao Zhang](https://md.research.elamur.ai/free/en/authors/biao-zhang/) and [Markus Freitag](https://md.research.elamur.ai/free/en/authors/markus-freitag/) · **Year:** 2025 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2025](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2025](https://md.research.elamur.ai/free/en/venues/findings/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.findings-acl.634/) · [PDF](https://aclanthology.org/2025.findings-acl.634.pdf) · [DOI 10.18653/v1/2025.findings-acl.634](https://doi.org/10.18653/v1/2025.findings-acl.634) · [OpenAlex](https://openalex.org/W4412888209)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [74](https://md.research.elamur.ai/free/en/papers/deutsch-2025-wmt24-expanding-language-coverage-wmt24/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Annual WMT test sets cover only about ten language pairs, which is too few to assess the multilingual translation ability of modern LLMs. This paper builds WMT24++, which reuses the 998 paragraph-level English sources of WMT24 (literary, news, social and speech domains). Professional translators produced new references and then post-edits for 46 additional languages and dialects, including Ukrainian (uk_UA). They also post-edited the existing references for 8 of the 9 original WMT24 targets. Together with Icelandic, the benchmark has 55 en→xx pairs. Native speakers spot-checked dialect fidelity. The authors scored 14 systems, both commercial MT services and LLMs, mainly with MetricX-24 and MetricX-24-QE. By these automatic scores, frontier LLMs ranked first for every language and even scored above the human translations. The authors stress that human evaluation is needed before claiming superhuman quality. The resource matters because it offers a recent, multi-domain, paragraph-level test set with wide language coverage, plus source-page screenshots useful for multimodal translation research.

## Key points
- **Coverage:** 55 English-to-X pairs.
  - 46 new languages/dialects have both references and post-edits.
  - 8 original WMT24 targets received post-edits; Icelandic got none because no vendor was available.
  - Regional variants include ar_EG/ar_SA, fr_CA/fr_FR, pt_BR/pt_PT, sw_KE/sw_TZ and zh_CN/zh_TW.
  - Data for bn_BD was dropped after failing the dialect check.
- **Data releases:** `google/wmt24pp` on Hugging Face, plus page screenshots in `google/wmt24pp-images`. Post-edits serve as the references for metric computation.
- **Systems evaluated:** Aya 23, Claude 3.5 Sonnet, Command R+, DeepL, Gemini-1.5 Pro/Flash, Google Translate, GPT-4o, OpenAI o1/o1-mini, Microsoft Translate, NLLB-54B, Tower-70B and Yandex.
  - 38 source segments (mostly URLs or emojis) were removed because they triggered verbose LLM outputs.
- **Main ranking result:** Under MetricX-24 significance clusters, OpenAI o1, Claude and Gemini had average ranks of 1.6, 1.9 and 2.1 across the 55 languages.
- **References vs. post-edits:** Post-editing raised or kept quality everywhere except ar_EG and ar_SA. For these two, the original references appear to have been written in Modern Standard Arabic rather than the target dialect, which the authors attribute to a vendor error.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Daniel Deutsch, Eleftheria Briakou, Isaac Rayburn Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, Shruti Rijhwani, Parker Riley, Elizabeth Salesky, Firas Trabelsi, Stephanie Winkler, Biao Zhang and Markus Freitag. 2025. WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects. Findings of the Association for Computational Linguistics: ACL 2025.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{deutsch2025wmt24,
  title = {{WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages \& Dialects}},
  author = {Daniel Deutsch, Eleftheria Briakou, Isaac Rayburn Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, Shruti Rijhwani, Parker Riley, Elizabeth Salesky, Firas Trabelsi, Stephanie Winkler, Biao Zhang and Markus Freitag},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2025},
  year = {2025},
  doi = {10.18653/v1/2025.findings-acl.634},
  url = {https://doi.org/10.18653/v1/2025.findings-acl.634}
}
```

- [← previous PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference](https://md.research.elamur.ai/free/en/papers/ji-2025-pku-saferlhf-multi-level-safety/)
- [next → GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/team-2025-v-glm-4-5v-glm/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/)

## References

42 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Duarte Miguel Alves, José Pombal, Nuno M Guerreiro, Pedro Henrique Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and Andre Martins. 2024. [Tower: An Open Multilingual Large Language Model for Translation-Related Tas…](https://md.research.elamur.ai/free/en/papers/alves-2024-tower-open-multilingual-large-language/)
2. Anthropic. 2024. Claude 3.5 Sonnet. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2024%2E+Claude+3%2E5+Sonnet%2E)
3. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019. [On the cross-lingual transferability of monolingual representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/). CoRR, abs/1910.11856.
4. Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Kelly Marchisio, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. 2024. [Aya 23: Open Weight Releases to Furt…](https://md.research.elamur.ai/free/en/papers/aryabumi-2024-aya-23-open-weight-releases/)
5. Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. [Findings of the 2019 Conference on Machine Translation (…](https://md.research.elamur.ai/free/en/papers/barrault-2019-findings-2019-conference-machine-translation/)
6. Eleftheria Briakou, Zhongtao Liu, Colin Cherry, and Markus Freitag. 2024. On the Implications of Verbose LLM Outputs: A Case Study in Translation Evaluation. Preprint, arXiv:2410.00863. [find](https://scholar.google.com/scholar?q=Eleftheria+Briakou%2C+Zhongtao+Liu%2C+Colin+Cherry%2C+and+Markus+Freitag%2E+2024%2E+On+the+Implications+of+Verbose+LLM+Outputs%3A+A+Case+Study+in+Translation+Evaluation%2E+Preprint%2C+arXiv%3A2410%2E00863%2E)
7. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/). Transactions of the Association for Computational Linguistics, 8:454–470…
8. Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No Language Left Behind: Scaling Human-Centered Machine Translation. arXiv preprint arXiv:2207.04672. [find](https://scholar.google.com/scholar?q=Marta+R+Costa%2DJuss%C3%A0%2C+James+Cross%2C+Onur+%C3%87elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+Jean+Maillard%2C+et+al%2E+2022%2E+No+Language+Left+Behind%3A+Scaling+H)
9. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255. [find](https://scholar.google.com/scholar?q=Jia+Deng%2C+Wei+Dong%2C+Richard+Socher%2C+Li%2DJia+Li%2C+Kai+Li%2C+and+Li+Fei%2DFei%2E+2009%2E+ImageNet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+2009+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+pa)
10. Chris Chinenye Emezue and Bonaventure F. P. Dossou. 2021. MMTAfrica: Multilingual Machine Translation for African Languages. In Proceedings of the Sixth Conference on Machine Translation, pages 398–411, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chris+Chinenye+Emezue+and+Bonaventure+F%2E+P%2E+Dossou%2E+2021%2E+MMTAfrica%3A+Multilingual+Machine+Translation+for+African+Languages%2E+In+Proceedings+of+the+Sixth+Conference+on+Machine+Translation%2C+pages+398%E2%80%9341)
11. Christian Federmann, Tom Kocmi, and Ying Xin. 2022. [NTREX-128 – News Test References for MT Evaluation of 128 Languages](https://md.research.elamur.ai/free/en/papers/federmann-2022-ntrex-128-news-test-references/). In Proceedings of the First Workshop on Scaling Up Multilingual Evaluation, pages 21–24, Online. Association for Computational Linguistics.
12. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://md.research.elamur.ai/free/en/papers/freitag-2021-experts-errors-context-large-scale/). Transactions of the Association for Computational Linguistics, 9:1460–1474.
13. Jay Gala, Pranjal A. Chitale, Raghavan AK, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar, Janki Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M. Khapra, Raj Dabre, and Anoop Kunchukuttan. 2023. [IndicTrans2: Towards High-Quality and Accessible Machine Translatio…](https://md.research.elamur.ai/free/en/papers/gala-2023-indictrans2-towards-high-quality-accessible/)
14. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2021. The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation. [find](https://scholar.google.com/scholar?q=Naman+Goyal%2C+Cynthia+Gao%2C+Vishrav+Chaudhary%2C+Peng%2DJen+Chen%2C+Guillaume+Wenzek%2C+Da+Ju%2C+Sanjana+Krishnan%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Francisco+Guzm%C3%A1n%2C+and+Angela+Fan%2E+2021%2E+The+FLORES%2D101+Evaluation+Benchmark)
15. Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2024. [XCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://md.research.elamur.ai/free/en/papers/guerreiro-2024-xcomet-transparent-machine-translation-evaluation/). Transactions of the Association for Computational Linguistics, 12:979–995.
16. Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam, Kazi Samin, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages. Preprint, arXiv:2106.13822. [find](https://scholar.google.com/scholar?q=Tahmid+Hasan%2C+Abhik+Bhattacharjee%2C+Md+Saiful+Islam%2C+Kazi+Samin%2C+Yuan%2DFang+Li%2C+Yong%2DBin+Kang%2C+M%2E+Sohel+Rahman%2C+and+Rifat+Shahriyar%2E+2021%2E+XL%2DSum%3A+Large%2DScale+Multilingual+Abstractive+Summarization+for+)
17. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://md.research.elamur.ai/free/en/papers/hu-2020-xtreme-massively-multilingual-multi-task/). CoRR, abs/2003.11080.
18. Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. [MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/juraska-2024-metricx-24-google-submission-wmt/). In Proceedings of the Ninth Conference on Machine Translation, pages 492–504, Miami, Florida, USA. Association for Computational Linguistics.
19. [Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, Martin Popel, Maja Popov…](https://md.research.elamur.ai/free/en/papers/kocmi-2024-findings-wmt24-general-machine-translation/)
20. Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2023. Pix2Struct: Screenshot parsing as pretraining for visual language understanding. In Proceedings of the 40th International Conf… [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Mandar+Joshi%2C+Iulia+Raluca+Turc%2C+Hexiang+Hu%2C+Fangyu+Liu%2C+Julian+Martin+Eisenschlos%2C+Urvashi+Khandelwal%2C+Peter+Shaw%2C+Ming%2DWei+Chang%2C+and+Kristina+Toutanova%2E+2023%2E+Pix2Struct%3A+Screenshot+par)
21. Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional Quality Metrics (MQM): A Framework for Declaring and Describing Translation Quality Metrics. Tradumàtica, (12):0455–463. [find](https://scholar.google.com/scholar?q=Arle+Lommel%2C+Hans+Uszkoreit%2C+and+Aljoscha+Burchardt%2E+2014%2E+Multidimensional+Quality+Metrics+%28MQM%29%3A+A+Framework+for+Declaring+and+Describing+Translation+Quality+Metrics%2E+Tradum%C3%A0tica%2C+%2812%29%3A0455%E2%80%93463%2E)
22. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. [Building a Large Annotated Corpus of English: The Penn Treebank](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/). Computational Linguistics, 19(2):313–330.
23. James Cross Onur Çelebi Maha Elbayad Kenneth Heafield Kevin Heffernan Elahe Kalbassi Janice Lam Daniel Licht Jean Maillard Anna Sun Skyler Wang Guillaume Wenzek Al Youngblood Bapi Akula Loic Barrault Gabriel Mejia Gonzalez Prangthip Hansanti John Hoffman Semarley Jarrett Kaushik Ram Sadagopan Dirk R… [find](https://scholar.google.com/scholar?q=James+Cross+Onur+%C3%87elebi+Maha+Elbayad+Kenneth+Heafield+Kevin+Heffernan+Elahe+Kalbassi+Janice+Lam+Daniel+Licht+Jean+Maillard+Anna+Sun+Skyler+Wang+Guillaume+Wenzek+Al+Youngblood+Bapi+Akula+Loic+Barrault+)
24. Eric W Noreen. 1989. Computer-Intensive Methods for Testing Hypotheses. Wiley New York. [find](https://scholar.google.com/scholar?q=Eric+W+Noreen%2E+1989%2E+Computer%2DIntensive+Methods+for+Testing+Hypotheses%2E+Wiley+New+York%2E)
25. OpenAI. 2024. Learning to Reason with LLMs. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2024%2E+Learning+to+Reason+with+LLMs%2E)
26. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a Method for Automatic Evaluation of Machine Translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
27. Maja Popović. 2015. [chrF: character n-gram F-score for automatic MT evaluation](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/). In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
28. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
29. Ricardo Rei, Nuno M. Guerreiro, José Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, José G. C. de Souza, and André Martins. 2023. [Scaling up CometKiwi: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task](https://md.research.elamur.ai/free/en/papers/rei-2023-scaling-up-cometkiwi-unbabel-ist/). In Proceedings of the Eighth Conference on Machine Translation, pages 841…
30. Ricardo Rei, Jose Pombal, Nuno M. Guerreiro, João Alves, Pedro Henrique Martins, Patrick Fernandes, Helena Wu, Tania Vaz, Duarte Alves, Amin Farajian, Sweta Agrawal, Antonio Farinhas, José G. C. De Souza, and André Martins. 2024. Tower v2: Unbabel-IST 2024 Submission for the General MT Shared Task.… [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Jose+Pombal%2C+Nuno+M%2E+Guerreiro%2C+Jo%C3%A3o+Alves%2C+Pedro+Henrique+Martins%2C+Patrick+Fernandes%2C+Helena+Wu%2C+Tania+Vaz%2C+Duarte+Alves%2C+Amin+Farajian%2C+Sweta+Agrawal%2C+Antonio+Farinhas%2C+Jos%C3%A9+G%2E+C%2E+De+So)
31. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.…
32. Sebastian Ruder, Jonathan Clark, Alexander Gutkin, Mihir Kale, Min Ma, Massimo Nicosia, Shruti Rijhwani, Parker Riley, Jean-Michel Sarr, Xinyi Wang, John Wieting, Nitish Gupta, Anna Katanova, Christo Kirov, Dana Dickinson, Brian Roark, Bidisha Samanta, Connie Tao, David Adelani, Vera Axelrod, Isaac… [find](https://scholar.google.com/scholar?q=Sebastian+Ruder%2C+Jonathan+Clark%2C+Alexander+Gutkin%2C+Mihir+Kale%2C+Min+Ma%2C+Massimo+Nicosia%2C+Shruti+Rijhwani%2C+Parker+Riley%2C+Jean%2DMichel+Sarr%2C+Xinyi+Wang%2C+John+Wieting%2C+Nitish+Gupta%2C+Anna+Katanova%2C+Christo+)
33. Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, and Desmond Elliott. 2023. Language modelling with pixels. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Phillip+Rust%2C+Jonas+F%2E+Lotz%2C+Emanuele+Bugliarello%2C+Elizabeth+Salesky%2C+Miryam+de+Lhoneux%2C+and+Desmond+Elliott%2E+2023%2E+Language+modelling+with+pixels%2E+In+The+Eleventh+International+Conference+on+Learning)
34. Ananya Sai B, Tanay Dixit, Vignesh Nagarajan, Anoop Kunchukuttan, Pratyush Kumar, Mitesh M. Khapra, and Raj Dabre. 2023. IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation Metrics for Indian Languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Ananya+Sai+B%2C+Tanay+Dixit%2C+Vignesh+Nagarajan%2C+Anoop+Kunchukuttan%2C+Pratyush+Kumar%2C+Mitesh+M%2E+Khapra%2C+and+Raj+Dabre%2E+2023%2E+IndicMT+Eval%3A+A+Dataset+to+Meta%2DEvaluate+Machine+Translation+Metrics+for+Indian)
35. Elizabeth Salesky, David Etter, and Matt Post. 2021. Robust open-vocabulary translation from visual text representations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7235–7252, Online and Punta Cana, Dominican Republic. Association for Computation… [find](https://scholar.google.com/scholar?q=Elizabeth+Salesky%2C+David+Etter%2C+and+Matt+Post%2E+2021%2E+Robust+open%2Dvocabulary+translation+from+visual+text+representations%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language)
36. Elizabeth Salesky, Philipp Koehn, and Matt Post. 2024. Benchmarking visually-situated translation of text in natural images. In Proceedings of the Ninth Conference on Machine Translation, pages 1167–1182, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Elizabeth+Salesky%2C+Philipp+Koehn%2C+and+Matt+Post%2E+2024%2E+Benchmarking+visually%2Dsituated+translation+of+text+in+natural+images%2E+In+Proceedings+of+the+Ninth+Conference+on+Machine+Translation%2C+pages+1167%E2%80%931)
37. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2022. [Language Models are Multilingual Chain-of-Thought Reasoners](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/). Preprint, arXiv:2210.03057.
38. Harman Singh, Nitish Gupta, Shikhar Bharadwaj, Dinesh Tewari, and Partha Talukdar. 2024a. Indic-GenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon… [find](https://scholar.google.com/scholar?q=Harman+Singh%2C+Nitish+Gupta%2C+Shikhar+Bharadwaj%2C+Dinesh+Tewari%2C+and+Partha+Talukdar%2E+2024a%2E+Indic%2DGenBench%3A+A+Multilingual+Benchmark+to+Evaluate+Generation+Capabilities+of+LLMs+on+Indic+Languages%2E+In+Pr)
39. [Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I. Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Wei-Yin Ko, Madeline Smith, Antoine Bosselut, Alice Oh, Andre F. T. Martins, Leshem Choshe…](https://md.research.elamur.ai/free/en/papers/singh-2025-global-mmlu-understanding-addressing-cultural/)
40. I Sutskever. 2014. Sequence to Sequence Learning with Neural Networks. arXiv preprint arXiv:1409.3215. [find](https://scholar.google.com/scholar?q=I+Sutskever%2E+2014%2E+Sequence+to+Sequence+Learning+with+Neural+Networks%2E+arXiv+preprint+arXiv%3A1409%2E3215%2E)
41. Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is All you Need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Neural Information Processing Systems.
42. Jiayi Wang, David Adelani, Sweta Agrawal, Marek Masiak, Ricardo Rei, Eleftheria Briakou, Marine Carpuat, Xuanli He, Sofia Bourhim, Andiswa Bukula, Muhidin Mohamed, Temitayo Olatoye, Tosin Adewumi, Hamam Mokayed, Christine Mwase, Wangui Kimotho, Foutse Yuehgoh, Anuoluwapo Aremu, Jessica Ojo, Shamsudd… [find](https://scholar.google.com/scholar?q=Jiayi+Wang%2C+David+Adelani%2C+Sweta+Agrawal%2C+Marek+Masiak%2C+Ricardo+Rei%2C+Eleftheria+Briakou%2C+Marine+Carpuat%2C+Xuanli+He%2C+Sofia+Bourhim%2C+Andiswa+Bukula%2C+Muhidin+Mohamed%2C+Temitayo+Olatoye%2C+Tosin+Adewumi%2C+Ham)

*Updated 24 October 2026*