---
title: "Bactrian-X — paper by Li et al. (2023)"
description: "Open instruction-tuned LLMs have mostly been trained on English data, and existing multilingual… 41 references, cited by 89 papers here."
authors: ["Haonan Li", "Fajri Koto", "Minghao Wu", "Alham Fikri Aji", "Timothy Baldwin"]
year: 2023
venue: "arXiv:2305.15011"
source: "https://arxiv.org/abs/2305.15011"
doi: "10.48550/arxiv.2305.15011"
arxiv: "2305.15011"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2023-bactrian-x-multilingual-replicable-instruction/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2023-bactrian-x-multilingual-replicable-instruction/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptation

# Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptation

**Authors:** [Haonan Li](https://md.research.elamur.ai/free/en/authors/haonan-li/), [Fajri Koto](https://md.research.elamur.ai/free/en/authors/fajri-koto/), [Minghao Wu](https://md.research.elamur.ai/free/en/authors/minghao-wu/), [Alham Fikri Aji](https://md.research.elamur.ai/free/en/authors/alham-fikri-aji/) and [Timothy Baldwin](https://md.research.elamur.ai/free/en/authors/timothy-baldwin/) · **Year:** 2023 · **Venue:** arXiv:2305.15011 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2305.15011) · [PDF](https://arxiv.org/pdf/2305.15011) · [DOI 10.48550/arxiv.2305.15011](https://doi.org/10.48550/arxiv.2305.15011) · [arXiv 2305.15011](https://arxiv.org/abs/2305.15011) · [OpenAlex](https://openalex.org/W4378465252)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [89](https://md.research.elamur.ai/free/en/papers/li-2023-bactrian-x-multilingual-replicable-instruction/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Open instruction-tuned LLMs have mostly been trained on English data, and existing multilingual resources such as xP3/xP3mt depend on fixed templates and classic NLP tasks rather than open-ended requests. To address this, the authors built Bactrian-X. They machine-translated the English instructions of Alpaca and Dolly into 51 further languages with the Google Translate API, excluding coding-related prompts. Instead of translating the answers, they had ChatGPT (gpt-3.5-turbo) respond directly to each translated instruction. They then trained lightweight LoRA adapters on top of LLaMA and BLOOM (7B and 13B). These adapters can be swapped in per language or language group without retraining the base model. The resulting BX LLaMA and BX BLOOM models were tested with zero-shot benchmarks, GPT-4-as-judge scoring and native-speaker rankings. In these tests they beat both the untuned base models and earlier instruction-tuned models such as Alpaca and BLOOMZ. The work matters because it supplies a large, openly released multilingual instruction corpus and a cheap, modular way to extend instruction-following to many languages.

## Key points
- **Dataset scale:** Bactrian-X contains about 67K instruction–response pairs for each of 52 languages, roughly 3.4M pairs in total. The languages follow the mBART-50 set. Translation cost about USD 10,000 and response generation about USD 3,000.
- **Translation quality:** Quality was checked by back-translating 100 samples per language. Mean BLEU was 48.1 and mean COMET was 90.2 (×100). Mongolian scored worst, with a BLEU of 28.
- **Response quality:** Human checks of the generated answers found over 80% good ratings for Arabic, Indonesian and Chinese. Tamil and Burmese answers were noticeably weaker.
- **Evaluation:** Benchmarks were XCOPA, XStoryCloze, XWinograd, EXAMS and the authors' new SentimentX. In human rankings, BX BLOOM led for ar, id, zh and ta, while BX LLaMA led for my and tl, which neither base model saw in pre-training.
- **Limitations:**
  - Instruction tuning alone cannot compensate for a language missing from pre-training.
  - Unseen languages are tokenized into 2–3× more tokens.
  - Context was capped at 768 sub-words.
  - Math and Fermi-style reasoning remained hard for all models.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The University of Melbourne](https://md.research.elamur.ai/free/en/universities/the-university-of-melbourne/) — University, Melbourne, Australia · 60 works · 21,199 citations
- [Monash University](https://md.research.elamur.ai/free/en/universities/monash-university/) — University, Melbourne, Australia · 31 works · 54,201 citations
- [Mohamed bin Zayed University of Artificial Intelligence](https://md.research.elamur.ai/free/en/universities/mohamed-bin-zayed-university-of-artificial-intelligence/) — University, Abu Dhabi, United Arab Emirates · 47 works · 4,456 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 467 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 77 works

## Cite

Cite as: Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji and Timothy Baldwin. 2023. Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptation. arXiv:2305.15011.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{li2023bactrian,
  title = {{Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptation}},
  author = {Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji and Timothy Baldwin},
  year = {2023},
  doi = {10.48550/arxiv.2305.15011},
  eprint = {2305.15011},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.15011}
}
```

- [← previous Inseq: An Interpretability Toolkit for Sequence Generation Models](https://md.research.elamur.ai/free/en/papers/sarti-2023-inseq-interpretability-toolkit-sequence-generation/)
- [next → Dictionary-based Phrase-level Prompting of Large Language Models for Machine Translation](https://md.research.elamur.ai/free/en/papers/ghazvininejad-2023-dictionary-based-phrase-level-prompting/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

41 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Nora Saleh Alturayeif, Hamzah Abdullah Luqman, and Moataz Aly Kamaleldin Ahmed. 2022. Mawqif: A multi-label Arabic dataset for target-specific stance detection. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 174–184, Abu Dhabi, United Arab Emirates (Hybr… [find](https://scholar.google.com/scholar?q=Nora+Saleh+Alturayeif%2C+Hamzah+Abdullah+Luqman%2C+and+Moataz+Aly+Kamaleldin+Ahmed%2E+2022%2E+Mawqif%3A+A+multi%2Dlabel+Arabic+dataset+for+target%2Dspecific+stance+detection%2E+In+Proceedings+of+the+The+Seventh+Arabi)
2. Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. [BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models](https://md.research.elamur.ai/free/en/papers/ben-zaken-2021-bitfit-simple-parameter-efficient-fine/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1–9, Dublin, Ireland. Asso…
3. [Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
4. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+and+Eric+P%2E+Xing%2E+2023%2E+Vicuna%3A+An+open%2Dsource+cha)
5. [Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi…](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
6. [Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang…](https://md.research.elamur.ai/free/en/papers/chung-2022-scaling-instruction-finetuned-language-models/)
7. Mike Conover, Matt Hayes, Ankit Mathur, Xiangrui Meng, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm. [https://www.databricks.com](https://www.databricks.com). [↗](https://www.databricks.com)
8. Demi Guo, Alexander Rush, and Yoon Kim. 2021. [Parameter-efficient transfer learning with diff pruning](https://md.research.elamur.ai/free/en/papers/guo-2021-parameter-efficient-transfer-learning-diff/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages…
9. Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslav Nakov. 2020. EXAMS: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Proc… [find](https://scholar.google.com/scholar?q=Momchil+Hardalov%2C+Todor+Mihaylov%2C+Dimitrina+Zlatkova%2C+Yoan+Dinkov%2C+Ivan+Koychev%2C+and+Preslav+Nakov%2E+2020%2E+EXAMS%3A+A+multi%2Dsubject+high+school+examinations+dataset+for+cross%2Dlingual+and+multilingual+que)
10. Yuta Hayashibe. 2020. Japanese realistic textual entailment corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6827–6834, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Yuta+Hayashibe%2E+2020%2E+Japanese+realistic+textual+entailment+corpus%2E+In+Proceedings+of+the+Twelfth+Language+Resources+and+Evaluation+Conference%2C+pages+6827%E2%80%936834%2C+Marseille%2C+France%2E+European+Language+Re)
11. [Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E…](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
12. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Ma… [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+De+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+NLP%2E+In+)
13. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations.
14. Fajri Koto, Afshin Rahimi, Jey Han Lau, and Timothy Baldwin. 2020. IndoLEM and IndoBERT: A benchmark dataset and pre-trained language model for Indonesian NLP. In Proceedings of the 28th International Conference on Computational Linguistics, pages 757–770, Barcelona, Spain (Online). International Co… [find](https://scholar.google.com/scholar?q=Fajri+Koto%2C+Afshin+Rahimi%2C+Jey+Han+Lau%2C+and+Timothy+Baldwin%2E+2020%2E+IndoLEM+and+IndoBERT%3A+A+benchmark+dataset+and+pre%2Dtrained+language+model+for+Indonesian+NLP%2E+In+Proceedings+of+the+28th+International)
15. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Brian+Lester%2C+Rami+Al%2DRfou%2C+and+Noah+Constant%2E+2021%2E+The+power+of+scale+for+parameter%2Defficient+prompt+tuning%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processing)
16. [Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyano…](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
17. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. 2022. Peft: State-of-the-art parameter-efficient fine-tuning methods. [https://github.com/huggingface/peft](https://github.com/huggingface/peft). [↗](https://github.com/huggingface/peft)
18. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Associati… [find](https://scholar.google.com/scholar?q=Nasrin+Mostafazadeh%2C+Nathanael+Chambers%2C+Xiaodong+He%2C+Devi+Parikh%2C+Dhruv+Batra%2C+Lucy+Vanderwende%2C+Pushmeet+Kohli%2C+and+James+Allen%2E+2016%2E+A+corpus+and+cloze+evaluation+for+deeper+understanding+of+commo)
19. [Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel…](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/)
20. Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Seid Muhie Yimam, David Ifeoluwa Adelani, Ibrahim Sa’id Ahmad, Nedjma Ousidhoum, Abinew Ali Ayele, Saif M. Mohammad, Meriem Beloucif, and Sebastian Ruder. 2023. SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval). In Proce… [find](https://scholar.google.com/scholar?q=Shamsuddeen+Hassan+Muhammad%2C+Idris+Abdulmumin%2C+Seid+Muhie+Yimam%2C+David+Ifeoluwa+Adelani%2C+Ibrahim+Sa%E2%80%99id+Ahmad%2C+Nedjma+Ousidhoum%2C+Abinew+Ali+Ayele%2C+Saif+M%2E+Mohammad%2C+Meriem+Beloucif%2C+and+Sebastian+Ruder)
21. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). abs/2303.08774.
22. [CoRR, Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
23. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
24. Edoardo M. Ponti, Goran Glava s, Olga Majewska, Qianchu Liu, Ivan Vuli’c, and Anna Korhonen. 2020. [XCOPA: A multilingual dataset for causal commonsense reasoning](https://md.research.elamur.ai/free/en/papers/ponti-2020-xcopa-multilingual-dataset-causal-commonsense/). arXiv preprint.
25. Maja Popović. 2017. [chrF++: words helping character n-grams](https://md.research.elamur.ai/free/en/papers/popovic-2017-chrf-words-helping-character-n/). In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics.
26. Matt Post. 2018. [A call for clarity in reporting BLEU scores](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/). In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics.
27. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. [COMET: A neural framework for MT evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.
28. [Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonatha…](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
29. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighof… [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+Jonathan+Tow%2C+Alexander+M%2E+Rush%2C+Stel)
30. Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. arXiv preprint arXiv:2008.00401. [find](https://scholar.google.com/scholar?q=Yuqing+Tang%2C+Chau+Tran%2C+Xian+Li%2C+Peng%2DJen+Chen%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Jiatao+Gu%2C+and+Angela+Fan%2E+2020%2E+Multilingual+translation+with+extensible+multilingual+pretraining+and+finetuning%2E+arXiv)
31. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca). [↗](https://github.com/tatsu-lab/stanford_alpaca)
32. Alexey Tikhonov and Max Ryabinin. 2021. [It’s All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning](https://md.research.elamur.ai/free/en/papers/tikhonov-2021-it-s-all-heads-attention/). In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3534–3546, Online. Association for Computational Linguistics.
33. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). CoRR, ab…
34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30.
35. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions. CoRR, abs/2212.10560. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A%2E+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2022%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+CoRR%2C+ab)
36. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Bosma%2C+Vincent+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+M%2E+Dai%2C+and+Quoc+V+Le%2E+2022%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+In+International+Conferenc)
37. Genta Indra Winata, Alham Fikri Aji, Samuel Cahyawijaya, Rahmad Mahendra, Fajri Koto, Ade Romadhony, Kemal Kurniawan, David Moeljadi, Radityo Eko Prasojo, Pascale Fung, Timothy Baldwin, Jey Han Lau, Rico Sennrich, and Sebastian Ruder. 2023. [NusaX: Multilingual parallel sentiment dataset for 10 Indon…](https://md.research.elamur.ai/free/en/papers/winata-2023-nusax-multilingual-parallel-sentiment-dataset/)
38. [Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho…](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
39. Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, and Alham Fikri Aji. 2023. Lamini-lm: A diverse herd of distilled models from large-scale instructions. CoRR, abs/2304.14402. [find](https://scholar.google.com/scholar?q=Minghao+Wu%2C+Abdul+Waheed%2C+Chiyu+Zhang%2C+Muhammad+Abdul%2DMageed%2C+and+Alham+Fikri+Aji%2E+2023%2E+Lamini%2Dlm%3A+A+diverse+herd+of+distilled+models+from+large%2Dscale+instructions%2E+CoRR%2C+abs%2F2304%2E14402%2E)
40. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. [mT5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling…
41. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023. [GLM-130b: An open bilingual pre-trained model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/). In The Eleventh…

*Updated 24 October 2026*