---
title: "Lost in Execution — paper by Luo et al. (2026)"
description: "Luo et al. introduce MLCL, a diagnostic benchmark built on the single-turn vanilla subset of the… 26 references, cited by 3 papers here."
authors: ["Zheng Luo", "T Pranav Kutralingam", "Ogochukwu N. Okoani", "Wanpeng Xu", "Hua Wei", "Xiyang Hu"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.2039/"
doi: "10.18653/v1/2026.acl-long.2039"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/luo-2026-lost-execution-multilingual-robustness-tool/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/luo-2026-lost-execution-multilingual-robustness-tool/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/5/) / Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models

# Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models

**Authors:** Zheng Luo, T Pranav Kutralingam, Ogochukwu N. Okoani, [Wanpeng Xu](https://md.research.elamur.ai/free/en/authors/wanpeng-xu/), [Hua Wei](https://md.research.elamur.ai/free/en/authors/hua-wei/) and [Xiyang Hu](https://md.research.elamur.ai/free/en/authors/xiyang-hu/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.2039/) · [PDF](https://aclanthology.org/2026.acl-long.2039.pdf) · [DOI 10.18653/v1/2026.acl-long.2039](https://doi.org/10.18653/v1/2026.acl-long.2039) · [OpenAlex](https://openalex.org/W7166871571)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 3 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Tool-calling benchmarks for LLM agents are almost entirely in English, so it is unclear how well models produce executable function calls when users write in other languages. Luo et al. introduce MLCL, a diagnostic benchmark built on the single-turn vanilla subset of the Berkeley Function Calling Leaderboard (BFCL). Its languages are Chinese and Hindi, plus Igbo as a low-resource case. The benchmark varies two independent factors: how much of the query is translated, and intent-preserving surface rewording. An error taxonomy separates execution-level violations from semantic mistakes.

The central finding is that models often pick the right tool and understand the request, yet the call still fails. The cause is that argument values are written in the user's language or script, while the interface expects English identifiers. The authors name this parameter value language mismatch. Simple inference-time fixes (explicit prompting, translating the query beforehand, translating arguments afterward) cut many of these errors. However, none of them reaches English-level accuracy, and some introduce semantic drift.

This matters for agents serving non-English users. Their reliability gaps stem largely from interface conventions, not only from weak language understanding.

## Key points
- **Benchmark design:** MLCL extends BFCL along two axes.
  - Query language composition: NT (not translated), PAR (partially translated), FT (fully translated).
  - Semantic perturbation: NO (none), PARA (paraphrase), SYNO (synonym replacement).
- **Dominant failure mode:** parameter value language mismatch. An example is passing "纽约市" or a Hindi transliteration instead of "New York City" as a location argument, which makes a correct call non-executable.
- **Language-dependent error profiles:**
  - For Chinese and Hindi, errors come mostly from tool-calling conventions and information lost in translation.
  - For Igbo, misreading the query itself accounts for a larger share of errors. As a result, mismatch-oriented fixes help less there.
- **Mitigation strategies tested:** PT (prompting to give English parameters), PRE (pre-translating the query), and POST (post-translating arguments), each performed by the tool-calling LLM itself. All three reduce errors, but none closes the gap to English performance.
- **Stated limitations:** single-turn setting only, three languages, an assumption that interfaces expect English parameter values, and deliberately simple mitigation baselines.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Southern California](https://md.research.elamur.ai/free/en/universities/university-of-southern-california/) — University, Los Angeles, United States · 162 works · 35,836 citations
- [Arizona State University](https://md.research.elamur.ai/free/en/universities/arizona-state-university/) — University, Tempe, United States · 50 works · 2,032 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works

## Cite

Cite as: Zheng Luo, T Pranav Kutralingam, Ogochukwu N. Okoani, Wanpeng Xu, Hua Wei and Xiyang Hu. 2026. Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{luo2026lost,
  title = {{Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models}},
  author = {Zheng Luo, T Pranav Kutralingam, Ogochukwu N. Okoani, Wanpeng Xu, Hua Wei and Xiyang Hu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2039},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2039}
}
```

- [← previous LongVideoAgent: Multi-Agent Reasoning with Long Videos](https://md.research.elamur.ai/free/en/papers/liu-2026-longvideoagent-multi-agent-reasoning-long/)
- [next → LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimization](https://md.research.elamur.ai/free/en/papers/tang-2026-lpo-accurate-gui-agent-interaction/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/5/)

## References

26 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Emre Can Acikgoz, Jeremiah Greer, Akul Datta, Ze Yang, William Zeng, Oussama Elachqar, Emmanouil Koukoumidis, Dilek Hakkani-Tür, and Gokhan Tur. 2025. [Can a single model master both multi-turn conversations and tool use? coalm: A unified conversational agentic language model](https://md.research.elamur.ai/free/en/papers/acikgoz-2025-can-single-model-master-both/). Preprint, arXiv:2502.08…
2. Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023. [MEGA: Multilingual evaluation of generative AI](https://md.research.elamur.ai/free/en/papers/ahuja-2023-mega-multilingual-evaluation-generative-ai/). In Proceedings of the 2023 Conference on Empirical Methods in N…
3. Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. 2023. [Fireact: Toward language agent fine-tuning](https://md.research.elamur.ai/free/en/papers/chen-2023-fireact-toward-language-agent-fine/). Preprint, arXiv:2310.05915.
4. Nuo Chen, Zinan Zheng, Ning Wu, Ming Gong, Dongmei Zhang, and Jia Li. 2024a. Breaking language barriers in multilingual mathematical reasoning: Insights and observations. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7001–7016, Miami, Florida, USA. Association for C… [find](https://scholar.google.com/scholar?q=Nuo+Chen%2C+Zinan+Zheng%2C+Ning+Wu%2C+Ming+Gong%2C+Dongmei+Zhang%2C+and+Jia+Li%2E+2024a%2E+Breaking+language+barriers+in+multilingual+mathematical+reasoning%3A+Insights+and+observations%2E+In+Findings+of+the+Associatio)
5. This work evaluates multilingual tool calling by LLMs under a controlled, English-only execution interface, using translated and perturbed versions of existing benchmark queries (Chinese, Hindi, Igbo) Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang… [find](https://scholar.google.com/scholar?q=This+work+evaluates+multilingual+tool+calling+by+LLMs+under+a+controlled%2C+English%2Donly+execution+interface%2C+using+translated+and+perturbed+versions+of+existing+benchmark+queries+%28Chinese%2C+Hindi%2C+Igbo%29)
6. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9510–9529. [find](https://scholar.google.com/scholar?q=Proceedings+of+the+62nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29%2C+pages+9510%E2%80%939529%2E)
7. Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. 2024c. [Agent-FLAN: Designing data and methods of effective agent tuning for large language models](https://md.research.elamur.ai/free/en/papers/chen-2024-agent-flan-designing-data-methods/). In Findings of the Association for Computational Linguistics: ACL 2024, pages 9354–9366, Bangkok,…
8. Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. 2025. Retool: Reinforcement learning for strategic tool use in llms. Preprint, arXiv:2504.11536. [find](https://scholar.google.com/scholar?q=Jiazhan+Feng%2C+Shijue+Huang%2C+Xingwei+Qu%2C+Ge+Zhang%2C+Yujia+Qin%2C+Baoquan+Zhong%2C+Chengquan+Jiang%2C+Jinxin+Chi%2C+and+Wanjun+Zhong%2E+2025%2E+Retool%3A+Reinforcement+learning+for+strategic+tool+use+in+llms%2E+Preprint)
9. Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021. [A survey of data augmentation approaches for nlp](https://md.research.elamur.ai/free/en/papers/feng-2021-survey-data-augmentation-approaches-nlp/). Preprint, arXiv:2105.03075.
10. Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. 2025. Toolrl: Reward is all tool learning needs. Preprint, arXiv:2504.13958. [find](https://scholar.google.com/scholar?q=Cheng+Qian%2C+Emre+Can+Acikgoz%2C+Qi+He%2C+Hongru+Wang%2C+Xiusi+Chen%2C+Dilek+Hakkani%2DT%C3%BCr%2C+Gokhan+Tur%2C+and+Heng+Ji%2E+2025%2E+Toolrl%3A+Reward+is+all+tool+learning+needs%2E+Preprint%2C+arXiv%3A2504%2E13958%2E)
11. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. 2024. [ToolLLM: Facilitating large language models to master 16000+ real…](https://md.research.elamur.ai/free/en/papers/qin-2023-toolllm-facilitating-large-language-models/)
12. Ella Rabinovich and Ateret Anaby Tavor. 2025. On the robustness of agentic function calling. In Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025), pages 298–304, Albuquerque, New Mexico. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ella+Rabinovich+and+Ateret+Anaby+Tavor%2E+2025%2E+On+the+robustness+of+agentic+function+calling%2E+In+Proceedings+of+the+5th+Workshop+on+Trustworthy+NLP+%28TrustNLP+2025%29%2C+pages+298%E2%80%93304%2C+Albuquerque%2C+New+Mexi)
13. Pankaj Kumar and Subhankar Mishra. 2025. Robustness in large language models: A survey of mitigation strategies and evaluation metrics. Preprint, arXiv:2505.18658. — [arXiv:2505.18658](https://arxiv.org/abs/2505.18658) [find](https://scholar.google.com/scholar?q=Pankaj+Kumar+and+Subhankar+Mishra%2E+2025%2E+Robustness+in+large+language+models%3A+A+survey+of+mitigation+strategies+and+evaluation+metrics%2E+Preprint%2C+arXiv%3A2505%2E18658%2E+%E2%80%94+%5BarXiv%3A2505%2E18658%5D%28https%3A%2F%2Farxiv%2Eo)
14. Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021. [XTREME-R: Towards more challenging and nuanced multilingual evaluation](https://md.research.elamur.ai/free/en/papers/ruder-2021-xtreme-r-more-challenging-nuanced/). In Proceedings of the 2021 Conference on Empirical Methods in Nat…
15. Xiaoou Liu, Tiejin Chen, Longchao Da, Chacha Chen, Zhen Lin, and Hua Wei. 2025. Uncertainty quantification and confidence calibration in [large language models: A survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/). In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 6107–6117.
16. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. [Toolformer: Language models can teach themselves to use tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/). Advances in Neural Information Processing Systems, 36:68539–68551.
17. Mehrad Moradshahi, Tianhao Shen, Kalika Bali, Monojit Choudhury, Gael de Chalendar, Anmol Goel, Sungkyun Kim, Prashant Kodali, Ponnurangam Kumaraguru, Nasredine Semmar, Sina Semnani, Jiwon Seo, Vivek Seshadri, Manish Shrivastava, Michael Sun, Aditya Yadavalli, Chaobin You, Deyi Xiong, and Monica Lam… [find](https://scholar.google.com/scholar?q=Mehrad+Moradshahi%2C+Tianhao+Shen%2C+Kalika+Bali%2C+Monojit+Choudhury%2C+Gael+de+Chalendar%2C+Anmol+Goel%2C+Sungkyun+Kim%2C+Prashant+Kodali%2C+Ponnurangam+Kumaraguru%2C+Nasredine+Semmar%2C+Sina+Semnani%2C+Jiwon+Seo%2C+Vivek+)
18. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. [Language models are multilingual chain-of-thought reasoners](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/). In The Eleventh International Conference on Learning Representa…
19. Aaron Parisi, Yao Zhao, and Noah Fiedel. 2022. [Talm: Tool augmented language models](https://md.research.elamur.ai/free/en/papers/parisi-2022-talm-tool-augmented-language-models/). Preprint, arXiv:2205.12255.
20. Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. 2023. [Mint: Evaluating llms in multi-turn interaction with tools and language feedback](https://md.research.elamur.ai/free/en/papers/wang-2023-mint-evaluating-llms-multi-turn/). arXiv preprint arXiv:2309.10691.
21. Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. 2025. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Shishir+G%2E+Patil%2C+Huanzhi+Mao%2C+Charlie+Cheng%2DJie+Ji%2C+Fanjia+Yan%2C+Vishnu+Suresh%2C+Ion+Stoica%2C+and+Joseph+E%2E+Gonzalez%2E+2025%2E+The+berkeley+function+calling+leaderboard+%28bfcl%29%3A+From+tool+use+to+agentic+eva)
22. Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2024. [Gorilla: Large language model connected with massive apis](https://md.research.elamur.ai/free/en/papers/patil-2023-gorilla-large-language-model-connected/). In Advances in Neural Information Processing Systems, volume 37, pages 126544–126565. Curran Associates, Inc.
23. Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. 2023. [Toolalpaca: Generalized tool learning for language models with 3000 simulated cases](https://md.research.elamur.ai/free/en/papers/tang-2023-toolalpaca-generalized-tool-learning-language/), 2023. URL https://arxiv.org/abs/2306.05301. [↗](https://arxiv.org/abs/2306.05301)
24. Yifan Wei, Xiaoyan Yu, Yixuan Weng, Tengfei Pan, Angsheng Li, and Li Du. 2025. Autotir: Autonomous tools integrated reasoning via reinforcement learning. Preprint, arXiv:2507.21836. — [arXiv:2507.21836](https://arxiv.org/abs/2507.21836) [find](https://scholar.google.com/scholar?q=Yifan+Wei%2C+Xiaoyan+Yu%2C+Yixuan+Weng%2C+Tengfei+Pan%2C+Angsheng+Li%2C+and+Li+Du%2E+2025%2E+Autotir%3A+Autonomous+tools+integrated+reasoning+via+reinforcement+learning%2E+Preprint%2C+arXiv%3A2507%2E21836%2E+%E2%80%94+%5BarXiv%3A2507%2E2183)
25. Mingyuan Wu, Jingcheng Yang, Jize Jiang, Meitang Li, Kaizhuo Yan, Hanchao Yu, Minjia Zhang, Chengxiang Zhai, and Klara Nahrstedt. 2025. Vtool-r1: Vlms learn to think with images via reinforcement learning on multimodal tool use. Preprint, arXiv:2505.19255. 44068 10 — [arXiv:2505.19255](https://arxiv.org/abs/2505.19255) [find](https://scholar.google.com/scholar?q=Mingyuan+Wu%2C+Jingcheng+Yang%2C+Jize+Jiang%2C+Meitang+Li%2C+Kaizhuo+Yan%2C+Hanchao+Yu%2C+Minjia+Zhang%2C+Chengxiang+Zhai%2C+and+Klara+Nahrstedt%2E+2025%2E+Vtool%2Dr1%3A+Vlms+learn+to+think+with+images+via+reinforcement+lear)
26. Jiayi Xu and Xiyang Hu. 2026. Language shapes mental health evaluations in large language models. Preprint, arXiv:2603.06910. — [arXiv:2603.06910](https://arxiv.org/abs/2603.06910) [find](https://scholar.google.com/scholar?q=Jiayi+Xu+and+Xiyang+Hu%2E+2026%2E+Language+shapes+mental+health+evaluations+in+large+language+models%2E+Preprint%2C+arXiv%3A2603%2E06910%2E+%E2%80%94+%5BarXiv%3A2603%2E06910%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2603%2E06910%29)

*Updated 31 October 2026*