---
title: "Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs"
description: "The authors propose the CIA (Cross Lingual Auto Evaluation) Suite. 31 references, cited by 10 papers here."
authors: ["Sumanth Doddapaneni", "Mohammed Safi Ur Rahman Khan", "Dilip Venkatesh", "Raj Dabre", "Anoop Kunchukuttan", "Mitesh M Khapra"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.1419/"
doi: "10.18653/v1/2025.acl-long.1419"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/doddapaneni-2025-cross-lingual-auto-evaluation-assessing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/doddapaneni-2025-cross-lingual-auto-evaluation-assessing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/) / Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs

# Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs

**Authors:** [Sumanth Doddapaneni](https://md.research.elamur.ai/free/en/authors/sumanth-doddapaneni/), [Mohammed Safi Ur Rahman Khan](https://md.research.elamur.ai/free/en/authors/mohammed-safi-ur-rahman-khan/), Dilip Venkatesh, [Raj Dabre](https://md.research.elamur.ai/free/en/authors/raj-dabre/), [Anoop Kunchukuttan](https://md.research.elamur.ai/free/en/authors/anoop-kunchukuttan/) and [Mitesh M Khapra](https://md.research.elamur.ai/free/en/authors/mitesh-m-khapra/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.1419/) · [PDF](https://aclanthology.org/2025.acl-long.1419.pdf) · [DOI 10.18653/v1/2025.acl-long.1419](https://doi.org/10.18653/v1/2025.acl-long.1419) · [OpenAlex](https://openalex.org/W4412889738)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [10](https://md.research.elamur.ai/free/en/papers/doddapaneni-2025-cross-lingual-auto-evaluation-assessing/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Automatic assessment of LLM outputs is mostly built around English. Other languages lack both challenging open-ended benchmarks and trustworthy evaluators, and reference answers in those languages are rare. The authors propose the CIA (Cross Lingual Auto Evaluation) Suite. In this setup, the prompt and the candidate response are in the target language, while the reference answer, rubric and grading instructions stay in English. The suite has three parts:

- **RECON**: a human-annotated test set of 500 prompts with human scores in six languages.
- **INTEL**: training data created by machine-translating Feedback-Collection.
- **HERCULE**: evaluators fine-tuned from Llama-3.1-8B that produce English feedback plus a score.

HERCULE matches human judgments more closely than large proprietary LLMs, with the biggest gains in low-resource languages. It also works zero-shot on languages it was not trained on. Ablations show that references help, that English references outperform translated ones, and that weight merging yields one workable multilingual judge. For under-resourced languages, this suggests a cheap way to evaluate models by reusing English references.

## Key points

- RECON covers Bengali, German, French, Hindi, Telugu and Urdu. Half of its 500 prompts (250) come from BigGenBench, which has per-example rubrics. It is meant both to benchmark multilingual LLMs and to meta-evaluate Evaluator LLMs.
- HERCULE models are reference-based evaluators trained on INTEL from the Llama-3.1-8B series. Starting from the instruction-tuned checkpoint speeds up convergence, and LoRA training also proved beneficial.
- Target-language references were impractical for Telugu and Bengali. Their tokenizer fertility exceeds 12, compared with under 3 for French and Hindi, so inputs overflowed the 4096-token limit and training failed to converge. With translated references, results were slightly worse for French and much worse for Hindi.
- Removing the reference answer entirely lowered evaluator quality on RECON.
- Average scores on RECON (Table 5) were 0.73 for per-language models, 0.69 for joint training, and 0.72 for both Linear and TIES merging. Merging matches individual models on high-resource languages but trails them on low-resource ones.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 325 works · 142,931 citations
- [Indian Institute of Technology Madras](https://md.research.elamur.ai/free/en/universities/indian-institute-of-technology-madras/) — University, Chennai, India · 16 works · 2,499 citations
- [National Institute of Information and Communications Technology](https://md.research.elamur.ai/free/en/labs/national-institute-of-information-and-communications-technology/) — Research institute, Tokyo, Japan · 23 works · 2,090 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works

## Cite

Cite as: Sumanth Doddapaneni, Mohammed Safi Ur Rahman Khan, Dilip Venkatesh, Raj Dabre, Anoop Kunchukuttan and Mitesh M Khapra. 2025. Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{doddapaneni2025cross,
  title = {{Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs}},
  author = {Sumanth Doddapaneni, Mohammed Safi Ur Rahman Khan, Dilip Venkatesh, Raj Dabre, Anoop Kunchukuttan and Mitesh M Khapra},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1419},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1419}
}
```

- [← previous Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMs](https://md.research.elamur.ai/free/en/papers/yang-2025-confidence-v-s-critique-decomposition/)
- [next → DiBiMT: A Gold Evaluation Benchmark for Studying Lexical Ambiguity in Machine Translation](https://md.research.elamur.ai/free/en/papers/martelli-2025-dibimt-gold-evaluation-benchmark-studying/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/)

## References

31 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Constable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
2. Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. [Alpacafarm: A simulation framework for methods that learn from human feedback](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/). In Advances in Neural Information Processing Systems 36: Annual Conferen…
3. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv: 2302.04166. [find](https://scholar.google.com/scholar?q=Jinlan+Fu%2C+See%2DKiong+Ng%2C+Zhengbao+Jiang%2C+and+Pengfei+Liu%2E+2023%2E+Gptscore%3A+Evaluate+as+you+desire%2E+arXiv+preprint+arXiv%3A+2302%2E04166%2E)
4. Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. 2024. [Arcee’s mergekit: A toolkit for merging large language models](https://md.research.elamur.ai/free/en/papers/goddard-2024-arcee-s-mergekit-toolkit-merging/). arXiv preprint arXiv:2403.13257.
5. Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, M. Choudhury, Kalika Bali, and Sunayana Sitaram. 2023. [Are large language model-based evaluators the solution to scaling up multilingual evaluation?](https://md.research.elamur.ai/free/en/papers/hada-2023-large-language-model-based-evaluators/) FINDINGS.
6. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020a. [Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization](https://md.research.elamur.ai/free/en/papers/hu-2020-xtreme-massively-multilingual-multi-task/). ArXiv, abs/2003.11080.
7. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020b. [XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization](https://md.research.elamur.ai/free/en/papers/hu-2020-xtreme-massively-multilingual-multi-task/). CoRR, abs/2003.11080.
8. Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. [Editing models with task arithmetic](https://md.research.elamur.ai/free/en/papers/ilharco-2022-editing-models-task-arithmetic/). In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
9. Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. 2023a. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. CoRR, abs/2310.11564. [find](https://scholar.google.com/scholar?q=Joel+Jang%2C+Seungone+Kim%2C+Bill+Yuchen+Lin%2C+Yizhong+Wang%2C+Jack+Hessel%2C+Luke+Zettlemoyer%2C+Hannaneh+Hajishirzi%2C+Yejin+Choi%2C+and+Prithviraj+Ammanabrolu%2E+2023a%2E+Personalized+soups%3A+Personalized+large+langua)
10. Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2023b. [Exploring the benefits of training expert language models over instruction tuning](https://md.research.elamur.ai/free/en/papers/jang-2023-exploring-benefits-training-expert-language/). In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawa…
11. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, S. Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2023a. [Prometheus: Inducing fine-grained evaluation capability in language models](https://md.research.elamur.ai/free/en/papers/kim-2023-prometheus-inducing-fine-grained-evaluation/). International Conference on Learning Representations.
12. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2023b. [Prometheus: Inducing fine-grained evaluation capability in language models](https://md.research.elamur.ai/free/en/papers/kim-2023-prometheus-inducing-fine-grained-evaluation/). CoRR, abs/2310.08491.
13. Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo… [find](https://scholar.google.com/scholar?q=Seungone+Kim%2C+Juyoung+Suk%2C+Ji+Yong+Cho%2C+Shayne+Longpre%2C+Chaeeun+Kim%2C+Dongkeun+Yoon%2C+Guijin+Son%2C+Yejin+Cho%2C+Sheikh+Shafayat%2C+Jinheon+Baek%2C+Sue+Hyun+Park%2C+Hyeonbin+Hwang%2C+Jinkyung+Jo%2C+Hyowon+Cho%2C+Haebin)
14. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024b. [Prometheus 2: An open source language model specialized in evaluating other language models](https://md.research.elamur.ai/free/en/papers/kim-2024-prometheus-2-open-source-language/). arXiv preprint arXiv: 2405.01535.
15. Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. 2022. Branch-train-merge: Embarrassingly parallel training of expert language models. CoRR, abs/2208.03306. [find](https://scholar.google.com/scholar?q=Margaret+Li%2C+Suchin+Gururangan%2C+Tim+Dettmers%2C+Mike+Lewis%2C+Tim+Althoff%2C+Noah+A%2E+Smith%2C+and+Luke+Zettlemoyer%2E+2022%2E+Branch%2Dtrain%2Dmerge%3A+Embarrassingly+parallel+training+of+expert+language+models%2E+CoRR%2C+)
16. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval). [↗](https://github.com/tatsu-lab/alpaca_eval)
17. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
18. Yang Liu, Dan Iter, Yichong Xu, Shuo Wang, Ruochen Xu, and Chenguang Zhu. 2023. [G-eval: Nlg evaluation using gpt-4 with better human alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/). Conference on Empirical Methods in Natural Language Processing.
19. YINGWEI MA, Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shanshan Li. 2024. At which training stage does code data help LLMs reasoning? In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=YINGWEI+MA%2C+Yue+Liu%2C+Yue+Yu%2C+Yuanliang+Zhang%2C+Yu+Jiang%2C+Changjian+Wang%2C+and+Shanshan+Li%2E+2024%2E+At+which+training+stage+does+code+data+help+LLMs+reasoning%3F+In+The+Twelfth+International+Conference+on+Le)
20. Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. 2022. Language models of code are few-shot commonsense learners. arXiv preprint arXiv: 2210.07128. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Yiming+Yang%2C+and+Graham+Neubig%2E+2022%2E+Language+models+of+code+are+few%2Dshot+commonsense+learners%2E+arXiv+preprint+arXiv%3A+2210%2E07128%2E)
21. Michael Matena and Colin Raffel. 2022. [Merging models with fisher-weighted averaging](https://md.research.elamur.ai/free/en/papers/matena-2021-merging-models-fisher-weighted-averaging/). In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.
22. Meta. 2024. Introducing meta llama 3: The most capable openly available llm to date. [https://ai.meta.com/blog/meta-llama-3/](https://ai.meta.com/blog/meta-llama-3/). Accessed: 2024-06-14. [↗](https://ai.meta.com/blog/meta-llama-3/)
23. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. [FActScore: Fine-grained atomic evaluation of factual precision in long form text generation](https://md.research.elamur.ai/free/en/papers/min-2023-factscore-fine-grained-atomic-evaluation/). In Proceedings of the 2023 Conference on Empirical Methods in Natural L…
24. OpenAI. 2024. Hello gpt-4 turbo. [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/). Accessed: 2024-10-14. [↗](https://openai.com/index/hello-gpt-4o/)
25. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
26. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [Squad: 100, 000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383–2392. The A…
27. Alexandre Ramé, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. 2023. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. In Advances in Neural Information Processing Systems 36: Annual… [find](https://scholar.google.com/scholar?q=Alexandre+Ram%C3%A9%2C+Guillaume+Couairon%2C+Corentin+Dancette%2C+Jean%2DBaptiste+Gaya%2C+Mustafa+Shukor%2C+Laure+Soulier%2C+and+Matthieu+Cord%2E+2023%2E+Rewarded+soups%3A+towards+pareto%2Doptimal+alignment+by+interpolating+wei)
28. Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. 2020. [COMET: A neural framework for MT evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 2685–2702. Association for Computational Linguisti…
29. Tetsuya Sakai. 2021. Evaluating evaluation measures for ordinal classification and ordinal quantification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pa… [find](https://scholar.google.com/scholar?q=Tetsuya+Sakai%2E+2021%2E+Evaluating+evaluation+measures+for+ordinal+classification+and+ordinal+quantification%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+an)
30. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. [BLEURT: Learning robust metrics for text generation](https://md.research.elamur.ai/free/en/papers/sellam-2020-bleurt-learning-robust-metrics-text/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics.
31. Harman Singh, Nitish Gupta, Shikhar Bharadwaj, Dinesh Tewari, and Partha Talukdar. 2024. Indicgenbench: A multilingual benchmark to evaluate generation capabilities of llms on indic languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… [find](https://scholar.google.com/scholar?q=Harman+Singh%2C+Nitish+Gupta%2C+Shikhar+Bharadwaj%2C+Dinesh+Tewari%2C+and+Partha+Talukdar%2E+2024%2E+Indicgenbench%3A+A+multilingual+benchmark+to+evaluate+generation+capabilities+of+llms+on+indic+languages%2E+In+Proc)

*Updated 31 October 2026*