---
title: "mCoT — paper by Lai and Nissim (2024) — summary and ideas"
description: "They define two pairwise metrics: one counts problems answered correctly in both languages, and… 47 references, cited by 29 papers here."
authors: ["Huiyuan Lai", "Malvina Nissim"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.649/"
doi: "10.18653/v1/2024.acl-long.649"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lai-2024-mcot-multilingual-instruction-tuning-reasoning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lai-2024-mcot-multilingual-instruction-tuning-reasoning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / mCoT: Multilingual Instruction Tuning for Reasoning Consistency in Language Models

# mCoT: Multilingual Instruction Tuning for Reasoning Consistency in Language Models

**Authors:** Huiyuan Lai and [Malvina Nissim](https://md.research.elamur.ai/free/en/authors/malvina-nissim/) · **Year:** 2024 · **Venue:** [Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.649/) · [PDF](https://aclanthology.org/2024.acl-long.649.pdf) · [DOI 10.18653/v1/2024.acl-long.649](https://doi.org/10.18653/v1/2024.acl-long.649) · [OpenAlex](https://openalex.org/W4402672000)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [29](https://md.research.elamur.ai/free/en/papers/lai-2024-mcot-multilingual-instruction-tuning-reasoning/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Chain-of-thought reasoning in large language models has been studied mostly in English, so it is unclear whether models reach the same answers when the same math problem is posed in other languages. Lai and Nissim frame this as **multilingual reasoning consistency**. They define two pairwise metrics: one counts problems answered correctly in both languages, and the other measures how often wrong answers coincide across a language pair.

They first test eight open-source LLMs on MGSM using few-shot CoT. The models perform very unevenly across languages, and they do worst on low-resource ones. To narrow this gap, the authors machine-translate English math reasoning data into ten further languages. This produces mCoT-MATH, a large instruction-tuning corpus, which they use to fine-tune a 7B model called mCoT.

On two multilingual benchmarks, mCoT is highly consistent across languages. It matches or beats both closed and open models, including much larger ones. The work matters because it shows that large-scale translated data can give under-represented languages compact, reliable reasoning models.

## Key points
- **mCoT-MATH** is described as the first large-scale multilingual math CoT dataset. Its roughly 6.3 million samples span 11 languages: English (EN) along with Spanish (ES), French (FR), German (DE), Russian (RU), Chinese (ZH), Japanese (JA), Thai (TH), Telugu (TE), Bengali (BN) and Swahili (SW).
- **Evaluation data:** MGSM contains 250 GSM8K test problems, translated by hand into 10 languages. GSM8K itself has about 7,500 training and 1,319 test problems. SW, BN, TE and TH each make up under 0.1% of PaLM's pretraining data.
- **Baseline models:** LLAMA2 (7B, 13B, 70B), Qwen (7B, 14B, 72B), Mistral-7B and Mixtral 8×7B. All were prompted 8-shot, except TE, which used 2-shot because of input length limits. The study also compared human-translated with machine-translated (Google Translate) prompts.
- **Metrics:** Correct consistency (CC) and incorrect consistency (IC) are computed for every pair of languages and are based only on final answers.
- **Release and limitations:** The data, code and model are available at github.com/laihuiyuan/mcot. The authors list as open problems the consistency of intermediate reasoning steps, filtering of machine-translated data, and reward learning.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Groningen](https://md.research.elamur.ai/free/en/universities/university-of-groningen/) — University, Groningen, Netherlands · 70 works · 5,142 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works

## Cite

Cite as: Huiyuan Lai and Malvina Nissim. 2024. mCoT: Multilingual Instruction Tuning for Reasoning Consistency in Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{lai2024mcot,
  title = {{mCoT: Multilingual Instruction Tuning for Reasoning Consistency in Language Models}},
  author = {Huiyuan Lai and Malvina Nissim},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.649},
  url = {https://doi.org/10.18653/v1/2024.acl-long.649}
}
```

- [← previous Language Varieties of Italy: Technology Challenges and Opportunities](https://md.research.elamur.ai/free/en/papers/ramponi-2024-language-varieties-italy-technology-challenges/)
- [next → Retrieval-Augmented Multilingual Knowledge Editing](https://md.research.elamur.ai/free/en/papers/wang-2024-retrieval-augmented-multilingual-knowledge-editing/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

47 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Se… — [PaLM 2 Technical Report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/)
2. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… — [Qwen Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
4. Oana-Maria Camburu, Brendan Shillingford, Pasquale Minervini, Thomas Lukasiewicz, and Phil Blunsom. 2020. Make up your mind! adversarial generation of inconsistent natural language explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4157– 4… [find](https://scholar.google.com/scholar?q=Oana%2DMaria+Camburu%2C+Brendan+Shillingford%2C+Pasquale+Minervini%2C+Thomas+Lukasiewicz%2C+and+Phil+Blunsom%2E+2020%2E+Make+up+your+mind%21+adversarial+generation+of+inconsistent+natural+language+explanations%2E+In+Pr)
5. Linzheng Chai, Jian Yang, Tao Sun, Hongcheng Guo, Jiaheng Liu, Bing Wang, Xiannian Liang, Jiaqi Bai, Tongliang Li, Qiyao Peng, and Zhoujun Li. 2024. xcot: Cross-lingual instruction tuning for cross-lingual chain-of-thought reasoning. arXiv preprint arXiv:2401.07037. — [arXiv:2401.07037](https://arxiv.org/abs/2401.07037) [find](https://scholar.google.com/scholar?q=Linzheng+Chai%2C+Jian+Yang%2C+Tao+Sun%2C+Hongcheng+Guo%2C+Jiaheng+Liu%2C+Bing+Wang%2C+Xiannian+Liang%2C+Jiaqi+Bai%2C+Tongliang+Li%2C+Qiyao+Peng%2C+and+Zhoujun+Li%2E+2024%2E+xcot%3A+Cross%2Dlingual+instruction+tuning+for+cross%2Dli)
6. Nuo Chen, Zinan Zheng, Ning Wu, Ming Gong, Yangqiu Song, Dongmei Zhang, and Jia Li. 2023a. Breaking language barriers in multilingual mathematical reasoning: Insights and observations. arXiv preprint arXiv:2310.20246. [find](https://scholar.google.com/scholar?q=Nuo+Chen%2C+Zinan+Zheng%2C+Ning+Wu%2C+Ming+Gong%2C+Yangqiu+Song%2C+Dongmei+Zhang%2C+and+Jia+Li%2E+2023a%2E+Breaking+language+barriers+in+multilingual+mathematical+reasoning%3A+Insights+and+observations%2E+arXiv+preprint+)
7. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023b. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research. [find](https://scholar.google.com/scholar?q=Wenhu+Chen%2C+Xueguang+Ma%2C+Xinyi+Wang%2C+and+William+W%2E+Cohen%2E+2023b%2E+Program+of+thoughts+prompting%3A+Disentangling+computation+from+reasoning+for+numerical+reasoning+tasks%2E+Transactions+on+Machine+Learnin)
8. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
9. Zheng Chuanyang, Liu Zhengying, Xie Enze, Li Zhenguo, and Li Yu. 2023. Progressive-hint prompting improves reasoning in large language models. arXiv preprint arXiv:2304.09797. [find](https://scholar.google.com/scholar?q=Zheng+Chuanyang%2C+Liu+Zhengying%2C+Xie+Enze%2C+Li+Zhenguo%2C+and+Li+Yu%2E+2023%2E+Progressive%2Dhint+prompting+improves+reasoning+in+large+language+models%2E+arXiv+preprint+arXiv%3A2304%2E09797%2E)
10. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
11. Song Jiang, Zahra Shakeri, Aaron Chan, Maziar Sanjabi, Hamed Firooz, Yinglong Xia, Bugra Akyildiz, Yizhou Sun, Jinchao Li, Qifan Wang, and Asli Celikyilmaz. 2024. Resprompt: Residual connection prompting advances multi-step reasoning in large language models. In The Twelfth International Conference… [find](https://scholar.google.com/scholar?q=Song+Jiang%2C+Zahra+Shakeri%2C+Aaron+Chan%2C+Maziar+Sanjabi%2C+Hamed+Firooz%2C+Yinglong+Xia%2C+Bugra+Akyildiz%2C+Yizhou+Sun%2C+Jinchao+Li%2C+Qifan+Wang%2C+and+Asli+Celikyilmaz%2E+2024%2E+Resprompt%3A+Residual+connection+prompt)
12. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
13. Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. 2023. GRACE: Discriminator-guided chain-of-thought reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15299–15328, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Muhammad+Khalifa%2C+Lajanugen+Logeswaran%2C+Moontae+Lee%2C+Honglak+Lee%2C+and+Lu+Wang%2E+2023%2E+GRACE%3A+Discriminator%2Dguided+chain%2Dof%2Dthought+reasoning%2E+In+Findings+of+the+Association+for+Computational+Linguistic)
14. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguistics, 9:1012–1031. [find](https://scholar.google.com/scholar?q=Yanai+Elazar%2C+Nora+Kassner%2C+Shauli+Ravfogel%2C+Abhilasha+Ravichander%2C+Eduard+Hovy%2C+Hinrich+Sch%C3%BCtze%2C+and+Yoav+Goldberg%2E+2021%2E+Measuring+and+improving+consistency+in+pretrained+language+models%2E+Transactio)
15. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). In Advances in Neural Information Processing Systems, volume 35, pages 22199–22213. Curran Associates, Inc.
16. Constanza Fierro and Anders Søgaard. 2022. Factual consistency of multilingual pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3046–3052, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Constanza+Fierro+and+Anders+S%C3%B8gaard%2E+2022%2E+Factual+consistency+of+multilingual+pretrained+language+models%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2022%2C+pages+3046%E2%80%933052%2C+Dubl)
17. Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023a. [Specializing smaller language models towards multi-step reasoning](https://md.research.elamur.ai/free/en/papers/fu-2023-specializing-smaller-language-models-towards/). In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, pages 10421–10430.
18. Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023b. [Complexity-based prompting for multi-step reasoning](https://md.research.elamur.ai/free/en/papers/fu-2022-complexity-based-prompting-multi-step/). In The Eleventh International Conference on Learning Representations.
19. Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. [Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes](https://md.research.elamur.ai/free/en/papers/hsieh-2023-distilling-step-step-outperforming-larger/). In Findings of the Association for Computati…
20. Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2023. [Large language models can self-improve](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1051–1068, Singapore. Association for Computational Linguistics.
21. Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. 2019. A logic-driven framework for consistency of neural models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),… [find](https://scholar.google.com/scholar?q=Tao+Li%2C+Vivek+Gupta%2C+Maitrey+Mehta%2C+and+Vivek+Srikumar%2E+2019%2E+A+logic%2Ddriven+framework+for+consistency+of+neural+models%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+)
22. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+International+Conference+on+Learning+Representations%2E)
23. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. [Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/). arXiv preprint arXiv:2308.09583.
24. Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2023. [Teaching small language models to reason](https://md.research.elamur.ai/free/en/papers/magister-2023-teaching-small-language-models-reason/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1773–1781, Toronto, Canada. Associa…
25. Eric Mitchell, Joseph Noh, Siyan Li, Will Armstrong, Ananth Agarwal, Patrick Liu, Chelsea Finn, and Christopher Manning. 2022. Enhancing self-consistency and performance of pre-trained language models through natural language inference. In Proceedings of the 2022 Conference on Empirical Methods in N… [find](https://scholar.google.com/scholar?q=Eric+Mitchell%2C+Joseph+Noh%2C+Siyan+Li%2C+Will+Armstrong%2C+Ananth+Agarwal%2C+Patrick+Liu%2C+Chelsea+Finn%2C+and+Christopher+Manning%2E+2022%2E+Enhancing+self%2Dconsistency+and+performance+of+pre%2Dtrained+language+models)
26. Ansong Ni, Jeevana Priya Inala, Chenglong Wang, Alex Polozov, Christopher Meek, Dragomir Radev, and Jianfeng Gao. 2023. Learning math reasoning from self-sampled correct and partially-correct solutions. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ansong+Ni%2C+Jeevana+Priya+Inala%2C+Chenglong+Wang%2C+Alex+Polozov%2C+Christopher+Meek%2C+Dragomir+Radev%2C+and+Jianfeng+Gao%2E+2023%2E+Learning+math+reasoning+from+self%2Dsampled+correct+and+partially%2Dcorrect+solution)
27. Sean O’Brien and Mike Lewis. 2023. [Contrastive decoding improves reasoning in large language models](https://md.research.elamur.ai/free/en/papers/o-brien-2023-contrastive-decoding-improves-reasoning-large/). arXiv preprint arXiv:2309.09117.
28. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
29. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. [Are NLP models really able to solve simple math word problems?](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/) In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, Online. Associat…
30. Jirui Qi, Raquel Fernández, and Arianna Bisazza. 2023. [Cross-lingual consistency of factual knowledge in multilingual language models](https://md.research.elamur.ai/free/en/papers/qi-2023-cross-lingual-consistency-factual-knowledge/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10650–10666, Singapore. Association for Computational Linguistics.
31. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, page 3505–3506, New York… [find](https://scholar.google.com/scholar?q=Jeff+Rasley%2C+Samyam+Rajbhandari%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+2020%2E+Deepspeed%3A+System+optimizations+enable+training+deep+learning+models+with+over+100+billion+parameters%2E+In+Proceedings+of+the+26t)
32. Abhilasha Ravichander, Eduard Hovy, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Cheung. 2020. [On the systematicity of probing contextualized word representations: The case of hypernymy in BERT](https://md.research.elamur.ai/free/en/papers/ravichander-2020-systematicity-probing-contextualized-word-representations/). In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, pages 88–102,…
33. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. [COMET: A neural framework for MT evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.
34. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. [Language models are multilingual chain-of-thought reasoners](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/). In The Eleventh International Conference on Learning Representa…
35. Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2023. [Distilling reasoning capabilities into smaller language models](https://md.research.elamur.ai/free/en/papers/shridhar-2023-distilling-reasoning-capabilities-into-smaller/). In Findings of the Association for Computational Linguistics: ACL 2023, pages 7059–7073, Toronto, Canada. Association for Computational Linguistics.
36. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
37. Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. [Solving math word problems with process- and outcome-based feedback](https://md.research.elamur.ai/free/en/papers/uesato-2022-solving-math-word-problems-process/). arXiv preprint arXiv:2211.14275.
38. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations.
39. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022a. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems.
40. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022b. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc.
41. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
42. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). In The Twelfth International Conference on Learning Representations.
43. Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2024. [MAmmoTH: Building math generalist models through hybrid instruction tuning](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/). In The Twelfth International Conference on Learning Representations.
44. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. [STar: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). In Advances in Neural Information Processing Systems.
45. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023. [Automatic chain of thought prompting in large language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-automatic-chain-thought-prompting-large/). In The Eleventh International Conference on Learning Representations.
46. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023. [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/). In The Eleventh International Conference on Learning Representation…
47. Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang. 2023. [Solving math word problems via cooperative reasoning induced language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-solving-math-word-problems-cooperative/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa…

*Updated 30 October 2026*