---
title: "TaCo — paper by Upadhayay and Behzadan (2023)"
description: "The authors propose a cheaper route. 31 references, cited by 14 papers here. Among them: 0."
authors: ["Bibek Upadhayay", "Vahid Behzadan"]
year: 2023
venue: "arXiv:2311.10797"
source: "https://arxiv.org/abs/2311.10797"
doi: "10.48550/arxiv.2311.10797"
arxiv: "2311.10797"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/upadhayay-2023-taco-enhancing-cross-lingual-transfer/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/upadhayay-2023-taco-enhancing-cross-lingual-transfer/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/13/) / TaCo: Enhancing Cross-Lingual Transfer for Low-Resource Languages in LLMs through Translation-Assisted Chain-of-Thought Processes

# TaCo: Enhancing Cross-Lingual Transfer for Low-Resource Languages in LLMs through Translation-Assisted Chain-of-Thought Processes

**Authors:** Bibek Upadhayay and Vahid Behzadan · **Year:** 2023 · **Venue:** arXiv:2311.10797 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2311.10797) · [PDF](https://arxiv.org/pdf/2311.10797) · [DOI 10.48550/arxiv.2311.10797](https://doi.org/10.48550/arxiv.2311.10797) · [arXiv 2311.10797](https://arxiv.org/abs/2311.10797) · [OpenAlex](https://openalex.org/W4388890640)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [14](https://md.research.elamur.ai/free/en/papers/upadhayay-2023-taco-enhancing-cross-lingual-transfer/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Making LLMs work in low-resource languages usually means expensive pretraining or full fine-tuning, and there are few multilingual benchmarks to measure progress. The authors propose a cheaper route. They machine-translated Alpaca-52K (GPT-4 version) and Dolly-15K into 132 languages with Google Cloud Translation, releasing the result as the Multilingual Instruction-Tuning Dataset (named MITS in the abstract and MITDS in the body). From these data they built TaCo (Translation-Assisted Cross-Linguality). Each training target walks through a chain of steps: render the non-English instruction in English, answer in English, then translate the answer back into the target language. Guanaco-33B, already tuned on OASST1, was further trained this way with LoRA adapters as a curriculum-learning step. Models for Nepali, Sanskrit, Maithili and Persian were judged by GPT-4 on a translated Vicuna Benchmark. On a low-resource language, TaCo reached a GPT-4 score of 82%, about twice that of plain instruction tuning. This suggests that strong English models can be extended to underserved languages at modest cost.

## Key points
- **Resources released:** translated Alpaca-52K, Dolly-15K and Vicuna Benchmark sets covering 132 languages, plus LoRA adapters for all four languages (github.com/UNHSAILLab/TaCo).
- **Training data:** the combined set has about 67K TaCo-formatted prompts. Translation quality was checked on 1,000 sentences per language using round-trip (en→xx→en) BLEU (SacreBLEU), ChrF and TER.
- **Training setup:** LoRA on q_proj, k_proj, v_proj and o_proj, with r=32, alpha=64 and dropout 0.05. The learning rate was 3e-4 and the cutoff length 2000.
- **Evaluation:** GPT-4 graded each answer individually on the multilingual Vicuna Benchmark. The headline result is 82% for a low-resource language, roughly double the instruction-tuning-only baseline.
- **Qualitative observation:** in the paper's Nepali examples, the model tuned without TaCo tends to fall into repeated sentence loops. The TaCo model gives a coherent answer in English and then in Nepali.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of New Haven](https://md.research.elamur.ai/free/en/universities/university-of-new-haven/) — University, West Haven, United States · 2 works · 17 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (3): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 882 works · [🌱 Low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/) · 287 works

## Cite

Cite as: Bibek Upadhayay and Vahid Behzadan. 2023. TaCo: Enhancing Cross-Lingual Transfer for Low-Resource Languages in LLMs through Translation-Assisted Chain-of-Thought Processes. arXiv:2311.10797.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{upadhayay2023taco,
  title = {{TaCo: Enhancing Cross-Lingual Transfer for Low-Resource Languages in LLMs through Translation-Assisted Chain-of-Thought Processes}},
  author = {Bibek Upadhayay and Vahid Behzadan},
  year = {2023},
  doi = {10.48550/arxiv.2311.10797},
  eprint = {2311.10797},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.10797}
}
```

- [← previous Prompting for explanations improves Adversarial NLI. Is this true? \Yes\ it is \true\ because ıt weakens superficial cues\](https://md.research.elamur.ai/free/en/papers/kavumba-2023-prompting-explanations-improves-adversarial-nli/)
- [next → Tailoring Self-Rationalizers with Multi-Reward Distillation](https://md.research.elamur.ai/free/en/papers/ramnath-2023-tailoring-self-rationalizers-multi-reward/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/13/)

## References

31 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403, 2023.
2. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. [Curriculum learning](https://md.research.elamur.ai/free/en/papers/bengio-2009-curriculum-learning/). In Proceedings of the 26th annual international conference on machine learning, pp. 41–48, 2009.
3. Angana Borah. Translationese: A brief introduction. [https://anganaborah.medium.com/](https://anganaborah.medium.com/) translationese-a-brief-introduction-db4575c0ee69, 2021. Accessed: 16-Sep-2022. [↗](https://anganaborah.medium.com/)
4. Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Bo Zheng, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. [XLM-E: Cross-lingual language model pre-training via ELECTRA](https://md.research.elamur.ai/free/en/papers/chi-2022-xlm-e-cross-lingual-language/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume…
5. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See [https://vicuna.lmsys](https://vicuna.lmsys). org (accessed 14 April 2023), 2023. [↗](https://vicuna.lmsys)
6. Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. Advances in neural information processing systems, 32, 2019. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Guillaume+Lample%2E+Cross%2Dlingual+language+model+pretraining%2E+Advances+in+neural+information+processing+systems%2C+32%2C+2019%2E)
7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. [Unsupervised cross-lingual representation learning at scale](https://md.research.elamur.ai/free/en/papers/conneau-2020-unsupervised-cross-lingual-representation-learning/). arXiv preprint arXiv:1911.02116, 2019.
8. Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023. URL [https://www.databricks.com/blog/2023/04/](https://www.databricks.com/blog/2023/04/) 12/dolly-first-open-commercially-viable-… [↗](https://www.databricks.com/blog/2023/04/)
9. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. [Qlora: Efficient finetuning of quantized llms](https://md.research.elamur.ai/free/en/papers/dettmers-2023-qlora-efficient-finetuning-quantized-llms/). arXiv preprint arXiv:2305.14314, 2023.
10. Philipp Dufter and Hinrich Schütze. Identifying elements essential for bert’s multilinguality. In Proceedings of the 2020 [Conference on Empirical Methods in Natural Language Processing (EMNLP), pp](https://md.research.elamur.ai/free/en/papers/dufter-2020-identifying-elements-essential-bert-s/). 4423–4437, 2020.
11. Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, and Alexis Conneau. [Larger-scale transformers for multilingual masked language modeling](https://md.research.elamur.ai/free/en/papers/goyal-2021-larger-scale-transformers-multilingual-masked/). arXiv preprint arXiv:2105.00572, 2021.
12. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300, 2020.
13. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). arXiv preprint arXiv:2106.09685, 2021.
14. Ayyoob ImaniGooghari, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André FT Martins, François Yvon, et al. Glot500: Scaling multilingual corpora and language models to 500 languages. arXiv preprint arXiv:2305.12182, 2023. [find](https://scholar.google.com/scholar?q=Ayyoob+ImaniGooghari%2C+Peiqin+Lin%2C+Amir+Hossein+Kargaran%2C+Silvia+Severini%2C+Masoud+Jalili+Sabet%2C+Nora+Kassner%2C+Chunlan+Ma%2C+Helmut+Schmid%2C+Andre%CC%81+FT+Martins%2C+Franc%CC%A7ois+Yvon%2C+et+al%2E+Glot500%3A+Scaling+multi)
15. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of naacL-HLT, volume 1, pp. 2, 2019.
16. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversations–democratizing large language model alignment. arXiv preprint arXiv:2304.[07327, 2023](https://md.research.elamur.ai/free/en/papers/kopf-2023-openassistant-conversations-democratizing-large-language/).
17. Moshe Koppel and Noam Ordan. [Translationese and its dialects](https://md.research.elamur.ai/free/en/papers/koppel-2011-translationese-its-dialects/). In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 1318–1326, 2011.
18. Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, et al. [Madlad-400: A multilingual and document-level large audited dataset](https://md.research.elamur.ai/free/en/papers/kudugunta-2023-madlad-400-multilingual-document-level/). arXiv preprint arXiv:2309.04662, 2023.
19. Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. Bactrian-x: A multilingual replicable instruction-following model with low-rank adaptation. arXiv preprint arXiv:2305.[15011, 2023](https://md.research.elamur.ai/free/en/papers/li-2023-bactrian-x-multilingual-replicable-instruction/).
20. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. [Crosslingual generalization through multitask finetuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/). arXiv preprint arXiv:2211.01786, 2022.
21. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023.
22. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. [Instruction tuning with gpt-4](https://md.research.elamur.ai/free/en/papers/peng-2023-instruction-tuning-gpt-4/). arXiv preprint arXiv:2304.03277, 2023.
23. Maja Popović. [chrf++: words helping character n-grams](https://md.research.elamur.ai/free/en/papers/popovic-2017-chrf-words-helping-character-n/). In Proceedings of the second conference on machine translation, pp. 612–618, 2017.
24. Matt Post. [A call for clarity in reporting bleu scores](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/). arXiv preprint arXiv:1804.08771, 2018.
25. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagne%CC%81%2C+Alexandra+Sasha+Luccioni%2C+Franc%CC%A7ois+Yvon%2C+Matthias+Galle%CC%81%2C+et+al%2E+Bloom%3A+A+176b%2Dparameter+op)
26. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. [Language models are multilingual chain-of-thought reasoners](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/). arXiv preprint arXiv:2210.03057, 2022.
27. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. [A study of translation edit rate with targeted human annotation](https://md.research.elamur.ai/free/en/papers/snover-2006-study-translation-edit-rate-targeted/). In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pp. 223–231, 2006. tloen. alpaca-lora…
28. UNESCO (IESALC). A decade to prevent the disappearance of 3,000 languages. [https://www.iesalc.unesco.org/en/2022/02/21/](https://www.iesalc.unesco.org/en/2022/02/21/) a-decade-to-prevent-the-disappearance-of-3000-languages/, 2022. Accessed: 2023-04-01. [↗](https://www.iesalc.unesco.org/en/2022/02/21/)
29. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). arXiv preprint arXiv:2206.07682, 2022a.
30. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837, 2022b.
31. Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, et al. [Polylm: An open source polyglot large language model](https://md.research.elamur.ai/free/en/papers/wei-2023-polylm-open-source-polyglot-large/). arXiv preprint arXiv:2307.06018, 2023. *Table 2: Translation Evaluation Metrics for Low Resources Languages* *Figure 2: Eval…

*Updated 2 November 2026*