---
title: "DiBiMT: A Gold Evaluation Benchmark for — Martelli 2025"
description: "Machine translation systems often mistranslate ambiguous words, especially when a word is used in one of its rarer senses."
authors: ["Federico Martelli", "Stefano Perrella", "Niccolò Campolungo", "Tina Munda", "Svetla Koeva", "Carole Tiberius", "Roberto Navigli"]
year: 2025
venue: "Computational Linguistics"
source: "https://aclanthology.org/2025.cl-2.1/"
doi: "10.1162/coli_a_00541"
licence: "CC BY-NC-ND 4.0 (Computational Linguistics, MIT Press)"
licence_url: "https://creativecommons.org/licenses/by-nc-nd/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/martelli-2025-dibimt-gold-evaluation-benchmark-studying/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/martelli-2025-dibimt-gold-evaluation-benchmark-studying/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2025](https://md.research.elamur.ai/free/papers/2025/) / DiBiMT: A Gold Evaluation Benchmark for Studying Lexical Ambiguity in Machine Translation

# DiBiMT: A Gold Evaluation Benchmark for Studying Lexical Ambiguity in Machine Translation

**Authors:** [Federico Martelli](https://md.research.elamur.ai/free/authors/federico-martelli/), [Stefano Perrella](https://md.research.elamur.ai/free/authors/stefano-perrella/), [Niccolò Campolungo](https://md.research.elamur.ai/free/authors/niccolo-campolungo/), Tina Munda, Svetla Koeva, Carole Tiberius, [Roberto Navigli](https://md.research.elamur.ai/free/authors/roberto-navigli/) · **Year:** 2025 · **Venue:** Computational Linguistics · **Licence:** [CC BY-NC-ND 4.0 (Computational Linguistics, MIT Press)](https://creativecommons.org/licenses/by-nc-nd/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.cl-2.1/) · [PDF](https://aclanthology.org/2025.cl-2.1.pdf) · [DOI 10.1162/coli\_a\_00541](https://doi.org/10.1162/coli_a_00541) · [OpenAlex](https://openalex.org/W4402985628)

**Citations:** 2 citations (openalex-snapshot · 2026-10-14) · [9](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Machine translation systems often mistranslate ambiguous words, especially when a word is used in one of its rarer senses. Earlier benchmarks for this problem were only partly built by hand and gave little insight into the kinds of errors models make. DiBiMT tackles both issues. It is a fully manually curated benchmark for studying disambiguation biases, meaning a system's tendency to fall back on a frequent but wrong sense, when translating from English into eight languages. The authors broaden an earlier five-language version (Campolungo et al. 2022) and manually extend and refine its lists of acceptable and unacceptable translations. They then evaluate commercial MT engines, open MT models and LLMs, and carry out a manual error analysis for every language pair. Further experiments look at the encoder's role in sense disambiguation, model capacity, multilinguality and beam search. Even the strongest systems remain far from perfect, and the authors show that standard MT evaluation setups fail to reveal these disambiguation errors. This makes a targeted testbed like DiBiMT necessary.

## Key points
- The eight target languages are Spanish, Slovene, Russian, Italian, German, Dutch, Chinese and Bulgarian. They cover the Balto-Slavic, Germanic, Italic and Sinitic subgroups.
- The earlier study tested seven systems. This version adds four more MT systems and eight LLMs:
  - Commercial engines: Google, DeepL, GPT-3.5 Turbo, GPT-4.
  - Open MT models: NLLB-200, M2M100, MBart50, OPUS.
  - Open LLMs: LLaMA2/3, Mistral, Gemma, Phi3, Tower.
- GPT-4 has the best average scores, with an SFI of about 64.4 and a PDI of about 68.8. Other averages:
  - DeepL: SFI about 54.7.
  - Gemma2-9B, the strongest open LLM: SFI about 60.4.
  - All systems combined: SFI about 38.7.
- Within the NLLB-200 family, scores climb steadily with model size. Multilingual OPUS performs clearly worse than the bilingual OPUS models.
- The benchmark is released at https://nlp.uniroma1.it/dibimt/.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/)

Topics: [Word sense disambiguation](https://md.research.elamur.ai/free/topics/word-sense-disambiguation/) · [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/)

## Cited by

Cited by

9 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 6 not yet.

- [Do Large Language Models Understand Word Senses?](https://md.research.elamur.ai/free/papers/lex-2025-meconi-llms-word-senses/)
- [Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics](https://md.research.elamur.ai/free/papers/perrella-2024-beyond-correlation-interpretable-evaluation-machine/)
- [Uncertainty Quantification for Evaluating Gender Bias in Machine Translation](https://md.research.elamur.ai/free/papers/staliunaite-2026-uncertainty-quantification-evaluating-gender-bias/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{martelli2025dibimt,
  title = {{DiBiMT: A Gold Evaluation Benchmark for Studying Lexical Ambiguity in Machine Translation}},
  author = {Martelli, Federico and Perrella, Stefano and Campolungo, Niccolò and Munda, Tina and Koeva, Svetla and Tiberius, Carole and Navigli, Roberto},
  journal = {Computational Linguistics},
  year = {2025},
  doi = {10.1162/coli\_a\_00541},
  url = {https://doi.org/10.1162/coli_a_00541}
}
```

- [← previous Diachronic Analysis of Phrasal Verbs in English Scientific Writing](https://md.research.elamur.ai/free/papers/alves-2025-diachronic-analysis-phrasal-verbs-english/)
- [next → Discourse structure and narration: A diachronic view from Germanic](https://md.research.elamur.ai/free/papers/bloom-2025-discourse-structure-narration-diachronic/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/papers/2025/)

## References

79 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abdin, Marah I., Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. Technical Report MSR-TR-2024-12, Microsoft. [↗](https://doi.org/10.48550/arxiv.2404.14219) [our card](https://md.research.elamur.ai/free/papers/abdin-2024-phi-3-technical-report-highly/)
2. Ainslie, Joshua, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. 2023. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901. https://doi.org/10.18653/v1/2023 .emnlp-main.298 [↗](https://doi.org/10.18653/v1/2023) [our card](https://md.research.elamur.ai/free/papers/ainslie-2023-gqa-training-generalized-multi-query/)
3. Alves, Duarte Miguel, José Pombal, Nuno M. Guerreiro, Pedro Henrique Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, et al. 2024. Tower: An open multilingual large language model for translation-related tasks. In Proceedings of First Conference on Language Modeling. [↗](https://doi.org/10.48550/arxiv.2402.17733) [our card](https://md.research.elamur.ai/free/papers/alves-2024-tower-open-multilingual-large-language/)
4. Barba, Edoardo, Luigi Procopio, and Roberto Navigli. 2021. ConSeC: Word sense disambiguation as continuous sense comprehension. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1492–1503. https://doi.org/10.18653/v1/2021 .emnlp-main.112 [↗](https://doi.org/10.18653/v1/2021)
5. Bawden, Rachel and François Yvon. 2023. Investigating the translation performance of a large multilingual language model: The case of BLOOM. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 157–170. [↗](https://doi.org/10.48550/arxiv.2303.01911)
6. Beltagy, Iz, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150. [↗](https://arxiv.org/abs/2004.05150)
7. Bevilacqua, Michele, Tommaso Pasini, Alessandro Raganato, and Roberto Navigli. 2021. Recent trends in word sense disambiguation: A survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 4330–4338. https://doi.org/10.24963/ijcai.2021/593 [↗](https://doi.org/10.24963/ijcai.2021/593)
8. Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, et al. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 33:1877–1901. [find](https://scholar.google.com/scholar?q=Brown%2C+Tom%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D%2E+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+et+al%2E+2020%2E+Language+mode)
9. Camacho-Collados, Jose and Roberto Navigli. 2017. BabelDomains: Large-scale domain labeling of lexical resources. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 223–228. https://doi.org/10.18653/v1/E17-2036 [↗](https://doi.org/10.18653/v1/e17-2036)
10. Campolungo, Niccolò, Federico Martelli, Francesco Saina, and Roberto Navigli. 2022. DiBiMT: A novel benchmark for measuring word sense disambiguation biases in machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4331–4352. https://doi.org/10.18653/v1/2022.acl -long.298 [↗](https://doi.org/10.18653/v1/2022.acl)
11. Conia, Simone and Roberto Navigli. 2021. Framing word sense disambiguation as a multi-label problem for model-agnostic knowledge integration. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3269–3275. https://doi.org/10.18653/v1/2021.eacl-main .286 [↗](https://doi.org/10.18653/v1/2021.eacl-main)
12. Conneau, Alexis, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451. https://doi.org/10.18653/v1/2020.acl -main.747 [↗](https://doi.org/10.18653/v1/2020.acl)
13. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. [↗](https://doi.org/10.18653/v1/n19-1423)
14. Dong, Qingxiu, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024. A survey on in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107–1128. [find](https://scholar.google.com/scholar?q=Dong%2C+Qingxiu%2C+Lei+Li%2C+Damai+Dai%2C+Ce+Zheng%2C+Jingyuan+Ma%2C+Rui+Li%2C+Heming+Xia%2C+Jingjing+Xu%2C+Zhiyong+Wu%2C+Baobao+Chang%2C+Xu+Sun%2C+Lei+Li%2C+and+Zhifang+Sui%2E+2024%2E+A+survey+on+in%2Dcontext+learning%2E+In+Proceedin) [our card](https://md.research.elamur.ai/free/papers/dong-2023-survey-context-learning/)
15. Elfwing, Stefan, Eiji Uchibe, and Kenji Doya. 2017. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. https://doi.org/10.1016/j.neunet.2017.12.012, PubMed: 29395652 [↗](https://doi.org/10.1016/j.neunet.2017.12.012)
16. Emelin, Denis, Ivan Titov, and Rico Sennrich. 2020. Detecting word sense disambiguation biases in machine translation for model-agnostic adversarial attacks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7635–7653. https://doi.org/10.18653/v1/2020 .emnlp-main.616 [↗](https://doi.org/10.18653/v1/2020)
17. Fan, Angela, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2021. Beyond English-Centric Multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48. [find](https://scholar.google.com/scholar?q=Fan%2C+Angela%2C+Shruti+Bhosale%2C+Holger+Schwenk%2C+Zhiyi+Ma%2C+Ahmed+El%2DKishky%2C+Siddharth+Goyal%2C+Mandeep+Baines%2C+Onur+Celebi%2C+Guillaume+Wenzek%2C+Vishrav+Chaudhary%2C+Naman+Goyal%2C+Tom+Birch%2C+Vitaliy+Liptchinsky%2C+)
18. Freitag, Markus, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460–1474. https://doi.org/10.1162/tacl a 00437 [↗](https://doi.org/10.1162/tacl_a_00437) [our card](https://md.research.elamur.ai/free/papers/freitag-2021-experts-errors-context-large-scale/)
19. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, et al. 2024. Gemini: A family of highly capable multimodal models. Technical report, Google DeepMind. [↗](https://doi.org/10.48550/arxiv.2312.11805) [our card](https://md.research.elamur.ai/free/papers/gemini-2023-team-gemini-family-highly-capable-multimodal/)
20. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024a. Gemma: Open models based on Gemini research and technology. Technical report, Google DeepMind. [↗](https://doi.org/10.48550/arxiv.2403.08295)
21. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024b. Gemma 2: Improving open language models at a practical size. Technical report, Google DeepMind. [↗](https://doi.org/10.48550/arxiv.2408.00118)
22. Gonzales, Annette Rios, Laura Mascarell, and Rico Sennrich. 2017. Improving word sense disambiguation in neural machine translation with sense embeddings. In Proceedings of the Second Conference on Machine Translation, pages 11–19. https://doi.org/10.18653/v1/W17-4702 [↗](https://aclanthology.org/W17-4702/) [our card](https://md.research.elamur.ai/free/papers/gonzales-2017-improving-word-sense-disambiguation-neural/)
23. Goyal, Naman, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522–538. https://doi.org/10.1162/tacl a 00474 [↗](https://doi.org/10.1162/tacl)
24. Graham, Yvette, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41. [find](https://scholar.google.com/scholar?q=Graham%2C+Yvette%2C+Timothy+Baldwin%2C+Alistair+Moffat%2C+and+Justin+Zobel%2E+2013%2E+Continuous+measurement+scales+in+human+evaluation+of+machine+translation%2E+In+Proceedings+of+the+7th+Linguistic+Annotation+Work)
25. Ide, Nancy and Keith Suderman. 2004. The American national corpus first release. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04). [find](https://scholar.google.com/scholar?q=Ide%2C+Nancy+and+Keith+Suderman%2E+2004%2E+The+American+national+corpus+first+release%2E+In+Proceedings+of+the+Fourth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9904%29%2E)
26. Iyer, Vivek, Edoardo Barba, Alexandra Birch, Jeff Z. Pan, and Roberto Navigli. 2023. Code-switching with word senses for pretraining in neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12889–12901. https://doi.org/10.18653/v1/2023.findings -emnlp.859 [↗](https://doi.org/10.18653/v1/2023.findings)
27. Iyer, Vivek, Pinzhen Chen, and Alexandra Birch. 2023. Towards effective disambiguation for machine translation with large language models. In Proceedings of the Eighth Conference on Machine Translation, pages 482–495. https://doi.org/10.18653/v1/2023 .wmt-1.44 [↗](https://doi.org/10.18653/v1/2023) [our card](https://md.research.elamur.ai/free/papers/iyer-2023-effective-disambiguation-machine-translation-large/)
28. Jiang, Albert Q., Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825. [↗](https://arxiv.org/abs/2310.06825)
29. Kendall, M. G. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81–93. https://doi.org/10.1093/biomet/30.1 -2.81 [↗](https://doi.org/10.1093/biomet/30.1)
30. Kocmi, Tom, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, et al. 2023. Findings of the 2023 conference on machine translation (WMT23): LLMs are here but not quite there yet. In Proceedings of the Eighth Conference on Machine Translation, pages 1–42. https://doi.org/10.18653/v1/2023.wmt-1.1 [↗](https://aclanthology.org/2023.wmt-1.1/)
31. Kocmi, Tom, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, et al. 2022. Findings of the 2022 conference on machine translation (WMT22). In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 1–45. https://doi.org/10.18653/v1/2023.wmt-1.1 [↗](https://aclanthology.org/2023.wmt-1.1/)
32. Koehn, Philipp. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79–86. [find](https://scholar.google.com/scholar?q=Koehn%2C+Philipp%2E+2005%2E+Europarl%3A+A+parallel+corpus+for+statistical+machine+translation%2E+In+Proceedings+of+Machine+Translation+Summit+X%3A+Papers%2C+pages+79%E2%80%9386%2E)
33. Krovetz, Robert and W. Bruce Croft. 1992. Lexical ambiguity and information retrieval. ACM Transactions on Information Systems (TOIS), 10(2):115–141. https://doi.org/10.1145/146802 .146810 [↗](https://doi.org/10.1145/146802)
34. Langone, Helen, Benjamin R. Haskell, and George A. Miller. 2004. Annotating WordNet. In Proceedings of the Workshop Frontiers in Corpus Annotation at HLT-NAACL 2004, pages 63–69. [find](https://scholar.google.com/scholar?q=Langone%2C+Helen%2C+Benjamin+R%2E+Haskell%2C+and+George+A%2E+Miller%2E+2004%2E+Annotating+WordNet%2E+In+Proceedings+of+the+Workshop+Frontiers+in+Corpus+Annotation+at+HLT%2DNAACL+2004%2C+pages+63%E2%80%9369%2E)
35. Lee, Katherine, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2018. Hallucinations in neural machine translation. In NeurIPS Workshop on Interpretability and Robustness in Audio, Speech, and Language (IRASL). [find](https://scholar.google.com/scholar?q=Lee%2C+Katherine%2C+Orhan+Firat%2C+Ashish+Agarwal%2C+Clara+Fannjiang%2C+and+David+Sussillo%2E+2018%2E+Hallucinations+in+neural+machine+translation%2E+In+NeurIPS+Workshop+on+Interpretability+and+Robustness+in+Audio%2C+S)
36. Lefever, Els and Veronique Hoste. 2010. SemEval-2010 Task 3: Cross-lingual word sense disambiguation. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 15–20. https://doi.org/10.3115/1621969.1621984 [↗](https://doi.org/10.3115/1621969.1621984)
37. Lefever, Els and Véronique Hoste. 2013. SemEval-2013 Task 10: Cross-lingual Word Sense Disambiguation. In Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 158–166. [find](https://scholar.google.com/scholar?q=Lefever%2C+Els+and+Ve%CC%81ronique+Hoste%2E+2013%2E+SemEval%2D2013+Task+10%3A+Cross%2Dlingual+Word+Sense+Disambiguation%2E+In+Second+Joint+Conference+on+Lexical+and+Computational+Semantics+%28%2A+SEM%29%2C+Volume+2%3A+Proceedings)
38. Ljubešić, Nikola and Kaja Dobrovoljc. 2019. What does neural bring? Analysing improvements in morphosyntactic annotation and lemmatisation of Slovenian, Croatian and Serbian. In Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing, pages 29–34. https://doi.org/10.18653/v1/W19-3704 [↗](https://doi.org/10.18653/v1/w19-3704)
39. Lommel, Arle, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional quality metrics (MQM): A framework for declaring and describing translation quality metrics. Tradumàtica, (12):455–463. https://doi.org/10.5565/rev/tradumatica.77 [↗](https://doi.org/10.5565/rev)
40. Maru, Marco, Simone Conia, Michele Bevilacqua, and Roberto Navigli. 2022. Nibbling at the hard core of Word Sense Disambiguation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4724–4737. https://doi.org/10.18653/v1/2022.acl -long.324 [↗](https://doi.org/10.18653/v1/2022.acl)
41. Marvin, Rebecca and Philipp Koehn. 2018. Exploring word sense disambiguation abilities of neural machine translation systems. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 125–131. [find](https://scholar.google.com/scholar?q=Marvin%2C+Rebecca+and+Philipp+Koehn%2E+2018%2E+Exploring+word+sense+disambiguation+abilities+of+neural+machine+translation+systems%2E+In+Proceedings+of+the+13th+Conference+of+the+Association+for+Machine+Trans)
42. McCarthy, Diana and Roberto Navigli. 2007. SemEval-2007 Task 10: English lexical substitution task. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007). https://doi.org/10.3115/1621474 .1621483 [↗](https://doi.org/10.3115/1621474)
43. Mihalcea, Rada, Ravi Sinha, and Diana McCarthy. 2010. SemEval-2010 Task 2: Cross-Lingual Lexical Substitution. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 9–14. [find](https://scholar.google.com/scholar?q=Mihalcea%2C+Rada%2C+Ravi+Sinha%2C+and+Diana+McCarthy%2E+2010%2E+SemEval%2D2010+Task+2%3A+Cross%2DLingual+Lexical+Substitution%2E+In+Proceedings+of+the+5th+International+Workshop+on+Semantic+Evaluation%2C+pages+9%E2%80%9314%2E)
44. Miller, George A. 1995. WordNet: A lexical database for English. Communications of the ACM, 38(11):39–41. https://doi.org/10 .1145/219717.219748 [↗](https://doi.org/10) [our card](https://md.research.elamur.ai/free/papers/lex-1994-miller-wordnet/)
45. Miller, George A., Claudia Leacock, Randee Tengi, and Ross T. Bunker. 1993. A semantic concordance. In Human Language Technology: Proceedings of a Workshop. https://doi.org/10.3115/1075671 .1075742 [↗](https://doi.org/10.3115/1075671)
46. Navigli, Roberto. 2009. Word sense disambiguation: A survey. ACM Computing Surveys (CSUR), 41(2):1–69. https://doi.org/10.1145/1459352 .1459355 [↗](https://doi.org/10.1145/1459352)
47. Navigli, Roberto, Michele Bevilacqua, Simone Conia, Dario Montagnini, and Francesco Cecconi. 2021. Ten years of BabelNet: A survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 4559–4567. https://doi.org/10 .24963/ijcai.2021/620 [↗](https://doi.org/10)
48. Navigli, Roberto and Simone Paolo Ponzetto. 2010. BabelNet: Building a very large multilingual semantic network. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 216–225. [find](https://scholar.google.com/scholar?q=Navigli%2C+Roberto+and+Simone+Paolo+Ponzetto%2E+2010%2E+BabelNet%3A+Building+a+very+large+multilingual+semantic+network%2E+In+Proceedings+of+the+48th+Annual+Meeting+of+the+Association+for+Computational+Linguist)
49. Neves, Mariana, Antonio Jimeno Yepes, Amy Siu, Roland Roller, Philippe Thomas, Maika Vicente Navarro, Lana Yeganova, Dina Wiemann, Giorgio Maria Di Nunzio, Federica Vezzani, Christel Gerardin, et al. 2022. Findings of the WMT 2022 biomedical translation shared task: Monolingual clinical case reports. In Proceedings of the Seventh Conference on Machine Translation, pages 694–723. [find](https://scholar.google.com/scholar?q=Neves%2C+Mariana%2C+Antonio+Jimeno+Yepes%2C+Amy+Siu%2C+Roland+Roller%2C+Philippe+Thomas%2C+Maika+Vicente+Navarro%2C+Lana+Yeganova%2C+Dina+Wiemann%2C+Giorgio+Maria+Di+Nunzio%2C+Federica+Vezzani%2C+Christel+Gerardin%2C+et+al%2E+)
50. Nivre, Joakim, Marie-Catherine De Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1659–1666. [find](https://scholar.google.com/scholar?q=Nivre%2C+Joakim%2C+Marie%2DCatherine+De+Marneffe%2C+Filip+Ginter%2C+Yoav+Goldberg%2C+Jan+Hajic%2C+Christopher+D%2E+Manning%2C+Ryan+McDonald%2C+Slav+Petrov%2C+Sampo+Pyysalo%2C+Natalia+Silveira%2C+Reut+Tsarfaty%2C+and+Daniel+Zeman) [our card](https://md.research.elamur.ai/free/papers/ud-2016-nivre-universal-dependencies-v1-multilingual-treebank/)
51. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207 .04672. [find](https://scholar.google.com/scholar?q=NLLB+Team%2C+Marta+R%2E+Costa%2Djussa%CC%80%2C+James+Cross%2C+Onur+C%CC%A7elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+Jean+Maillard%2C+et+al%2E+2022%2E+No+language+left+beh)
52. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2024. GPT-4 technical report. Technical report, OpenAI. [↗](https://doi.org/10.4230/lipics.cosit.2024.11) [our card](https://md.research.elamur.ai/free/papers/openai-2023-gpt-4-technical-report/)
53. Orlando, Riccardo, Simone Conia, Fabrizio Brignone, Francesco Cecconi, and Roberto Navigli. 2021. AMuSE-WSD: An all-in-one multilingual system for easy Word Sense Disambiguation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 298–307. https://doi.org/10 .18653/v1/2021.emnlp-demo.34 [↗](https://aclanthology.org/2021.emnlp-demo.34/)
54. Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. https://doi.org/10.3115/1073083 .1073135 [↗](https://doi.org/10.3115/1073083) [our card](https://md.research.elamur.ai/free/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
55. Paul, Douglas B. and Janet M. Baker. 1992. The design for the Wall Street Journal-based CSR corpus. In Speech and Natural Language: Proceedings of a Workshop. https://doi.org/10.3115/1075527 .1075614 [↗](https://doi.org/10.3115/1075527)
56. Popović, Maja. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395. https://doi.org/10 .18653/v1/W15-3049 [↗](https://aclanthology.org/W15-3049/) [our card](https://md.research.elamur.ai/free/papers/popovic-2015-chrf-character-n-gram-f/)
57. Popović, Maja. 2017. chrF++: Words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618. https://doi.org/10 .18653/v1/W17-4770 [↗](https://aclanthology.org/W17-4770/) [our card](https://md.research.elamur.ai/free/papers/popovic-2017-chrf-words-helping-character-n/)
58. Post, Matt. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191. https://doi.org/10.18653/v1/W18-6319 [↗](https://aclanthology.org/W18-6319/) [our card](https://md.research.elamur.ai/free/papers/post-2018-call-clarity-reporting-bleu-scores/)
59. Pradhan, Sameer, Edward Loper, Dmitriy Dligach, and Martha Palmer. 2007. [find](https://scholar.google.com/scholar?q=Pradhan%2C+Sameer%2C+Edward+Loper%2C+Dmitriy+Dligach%2C+and+Martha+Palmer%2E+2007%2E)
60. SemEval-2007 Task-17: English lexical sample, SRL and all words. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 87–92. https://doi.org/10.3115/1621474.1621490 [↗](https://doi.org/10.3115/1621474.1621490)
61. Qi, Peng, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101–108. https://doi.org/10 .18653/v1/2020.acl-demos.14 [↗](https://aclanthology.org/2020.acl-demos.14/) [our card](https://md.research.elamur.ai/free/papers/ud-2020-qi-stanza-python-natural-language-processing/)
62. Raganato, Alessandro, Jose Camacho-Collados, and Roberto Navigli. 2017. Word sense disambiguation: A unified evaluation framework and empirical comparison. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 99–110. https://doi.org/10.18653/v1/E17-1010 [↗](https://aclanthology.org/E17-1010/) [our card](https://md.research.elamur.ai/free/papers/onto-2017-raganato-word-sense-disambiguation-unified-evaluation/)
63. Raganato, Alessandro, Yves Scherrer, and Jörg Tiedemann. 2019. The MuCoW test suite at WMT 2019: Automatically harvested multilingual contrastive word sense disambiguation test sets for machine translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 470–480. https://doi.org/10.18653/v1/W19-5354 [↗](https://aclanthology.org/W19-5354/)
64. Rei, Ricardo, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C. Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585. [find](https://scholar.google.com/scholar?q=Rei%2C+Ricardo%2C+Jose%CC%81+G%2E+C%2E+de+Souza%2C+Duarte+Alves%2C+Chrysoula+Zerva%2C+Ana+C%2E+Farinha%2C+Taisiya+Glushkova%2C+Alon+Lavie%2C+Luisa+Coheur%2C+and+Andre%CC%81+F%2E+T%2E+Martins%2E+2022%2E+COMET%2D22%3A+Unbabel%2DIST+2022+submission+fo) [our card](https://md.research.elamur.ai/free/papers/rei-2022-comet-22-unbabel-ist-2022/)
65. Rios, Annette, Mathias Müller, and Rico Sennrich. 2018. The word sense disambiguation test suite at WMT18. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 588–596. https://doi.org/10 .18653/v1/W18-6437 [↗](https://aclanthology.org/W18-6437/)
66. Shazeer, Noam. 2019. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150. [↗](https://arxiv.org/abs/1911.02150)
67. Shazeer, Noam. 2020. GLU variants improve transformer. arXiv preprint arXiv:2002.05202. [↗](https://arxiv.org/abs/2002.05202)
68. Su, Jianlin, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063. https://doi.org/10.1016/j.neucom.2023.127063 [↗](https://doi.org/10.1016/j.neucom.2023.127063)
69. Tang, Yuqing, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2021. Multilingual translation from denoising pre-training. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3450–3466. https://doi.org/10.18653/v1/2021.findings-acl.304 [↗](https://doi.org/10.18653/v1/2021.findings-acl.304) [our card](https://md.research.elamur.ai/free/papers/tang-2021-multilingual-translation-denoising-pre-training/)
70. Tiedemann, Jörg. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2214–2218. [find](https://scholar.google.com/scholar?q=Tiedemann%2C+Jo%CC%88rg%2E+2012%2E+Parallel+data%2C+tools+and+interfaces+in+OPUS%2E+In+Proceedings+of+the+Eighth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9912%29%2C+pages+2214%E2%80%932218%2E)
71. Tiedemann, Jörg and Santhosh Thottingal. 2020. OPUS-MT – Building open translation services for the World. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation (EAMT). [find](https://scholar.google.com/scholar?q=Tiedemann%2C+Jo%CC%88rg+and+Santhosh+Thottingal%2E+2020%2E+OPUS%2DMT+%E2%80%93+Building+open+translation+services+for+the+World%2E+In+Proceedings+of+the+22nd+Annual+Conference+of+the+European+Association+for+Machine+Transla)
72. Touvron, Hugo, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. [↗](https://arxiv.org/abs/2302.13971)
73. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, 30:6000–6010. [find](https://scholar.google.com/scholar?q=Vaswani%2C+Ashish%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Proceedings+of+the+31st+International)
74. Virtanen, Pauli, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. 2020. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261–272. https://doi.org/10.1038/s41592-020-0772-5 [↗](https://doi.org/10.1038/s41592-019-0686-2) [our card](https://md.research.elamur.ai/free/papers/virtanen-2019-scipy-1-0-fundamental-algorithms/)
75. Voita, Elena, Rico Sennrich, and Ivan Titov. 2021. Language modeling, lexical translation, reordering: The training process of NMT through the lens of classical SMT. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8478–8491. https://doi.org/10.18653/v1/2021 .emnlp-main.667 [↗](https://doi.org/10.18653/v1/2021)
76. Xu, Haoran, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2024. A paradigm shift in machine translation: Boosting translation performance of large language models. In Proceedings of The Twelfth International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.2309.11674)
77. Zhang, Biao, Barry Haddow, and Alexandra Birch. 2023. Prompting large language model for machine translation: A case study. In Proceedings of the 40th International Conference on Machine Learning, pages 41092–41110. [↗](https://doi.org/10.48550/arxiv.2301.07069) [our card](https://md.research.elamur.ai/free/papers/zhang-2023-prompting-large-language-model-machine/)
78. Zhang, Tianyi, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In Proceedings of the International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.1904.09675)
79. Zhu, Wenhao, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2024. Multilingual machine translation with large language models: Empirical results and analysis. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2765–2781. https://doi.org/10.18653/v1/2024.findings -naacl.176 [↗](https://doi.org/10.18653/v1/2024.findings) [our card](https://md.research.elamur.ai/free/papers/zhu-2024-multilingual-machine-translation-large-language/)