---
title: "Specializing Smaller Language Models towards — Fu 2023"
description: "Chain-of-thought math reasoning is usually thought to appear only in models above roughly 100B parameters, which puts it out of reach for most researchers."
authors: ["Yao Fu", "Hao Peng", "Litu Ou", "Ashish Sabharwal", "Tushar Khot"]
year: 2023
venue: "arXiv:2301.12726"
source: "https://arxiv.org/abs/2301.12726"
doi: "10.48550/arxiv.2301.12726"
arxiv: "2301.12726"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/fu-2023-specializing-smaller-language-models-towards/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/fu-2023-specializing-smaller-language-models-towards/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / Specializing Smaller Language Models towards Multi-Step Reasoning

# Specializing Smaller Language Models towards Multi-Step Reasoning

**Authors:** [Yao Fu](https://md.research.elamur.ai/free/en/authors/yao-fu/), [Hao Peng](https://md.research.elamur.ai/free/en/authors/hao-peng/), Litu Ou, [Ashish Sabharwal](https://md.research.elamur.ai/free/en/authors/ashish-sabharwal/), [Tushar Khot](https://md.research.elamur.ai/free/en/authors/tushar-khot/) · **Year:** 2023 · **Venue:** arXiv:2301.12726 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2301.12726) · [PDF](https://arxiv.org/pdf/2301.12726) · [DOI 10.48550/arxiv.2301.12726](https://doi.org/10.48550/arxiv.2301.12726) · [arXiv 2301.12726](https://arxiv.org/abs/2301.12726) · [OpenAlex](https://openalex.org/W4318719086)

**Citations:** 43 citations (openalex-snapshot · 2026-10-14) · [179](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Chain-of-thought math reasoning is usually thought to appear only in models above roughly 100B parameters, which puts it out of reach for most researchers. The authors argue that a small model cannot be good at everything, but it can do well on one task if its limited capacity goes there. To test this, they distill step-by-step solutions to GSM8K problems from GPT-3.5 (code-davinci-002) into FlanT5 and T5 students of up to 11B parameters. Teacher and student tokenizers are aligned with dynamic programming, and the training data mixes in-context and zero-shot prompt formats. Checkpoints are chosen by their scores on held-out math datasets, not on GSM8K validation. Specialized FlanT5 models with 250M, 760M and 3B parameters gain about 10 accuracy points on average across four math benchmarks. The cost is a large loss of general ability on BigBench Hard. After specialization, accuracy grows log-linearly with model size, which challenges the idea that CoT is purely emergent. For teams with limited compute, this shows a workable way to build compact, task-specific reasoners.

## Key points
- **Setup:** the teacher is code-davinci-002 and the students are FlanT5 and T5. Training uses GSM8K only. MultiArith, ASDiv and SVAMP test out-of-distribution generalization, and BigBench Hard (BBH) measures how much general ability is lost.
- **Trade-off:** soon after tuning starts, the models lose all of their BBH chain-of-thought prompting ability, and over time a large part of their answer-only BBH ability. Instruction-tuned FlanT5 ends up stronger than specialized raw T5, although T5 improves more relative to its starting point.
- **Model selection:** late in training, GSM8K and out-of-distribution scores move independently. Choosing checkpoints on the combined MultiArith/ASDiv/SVAMP set gives better OOD results and costs less on GSM8K.
- **Data format and distillation:** training on in-context examples gives both few-shot and zero-shot skill. Training on zero-shot data only gradually removes in-context learning. Matching the teacher's token distributions (KL, top-5 probabilities) converges faster than training on sampled reasoning paths, with similar final accuracy.
- **Data needs:** FlanT5 3B levels off at about 90K examples, while the Base model keeps improving through three epochs of about 130K examples each.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/)

## Cited by

Cited by

179 papers of our library cite it (of the 140884 whose bibliographies we read); 9 of them have a page here, 170 not yet.

- [ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs](https://md.research.elamur.ai/free/en/papers/chen-2024-reconcile-round-table-conference-improves/)
- [ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving](https://md.research.elamur.ai/free/en/papers/gou-2023-tora-tool-integrated-reasoning-agent/)
- [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-trustworthy-llms-survey-guideline-evaluating/)
- [Reasoning with Language Model Prompting: A Survey](https://md.research.elamur.ai/free/en/papers/qiao-2022-reasoning-language-model-prompting-survey/)
- [Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents](https://md.research.elamur.ai/free/en/papers/sun-2023-chatgpt-good-search-investigating-large/)
- [Reason-to-Learn (R2L): Multi-Agent Knowledge Distillation for Lightweight LLMs in Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/tu-2026-reason-learn-r2l-multi-agent/)
- [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/)
- [Scaling Relationship on Learning Mathematical Reasoning with Large Language Models](https://md.research.elamur.ai/free/en/papers/yuan-2023-scaling-relationship-learning-mathematical-reasoning/)
- [A Survey on Model Compression for Large Language Models](https://md.research.elamur.ai/free/en/papers/zhu-2024-survey-model-compression-large-language/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{fu2023specializing,
  title = {{Specializing Smaller Language Models towards Multi-Step Reasoning}},
  author = {Fu, Yao and Peng, Hao and Ou, Litu and Sabharwal, Ashish and Khot, Tushar},
  year = {2023},
  doi = {10.48550/arxiv.2301.12726},
  eprint = {2301.12726},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2301.12726}
}
```

- [← previous Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4](https://md.research.elamur.ai/free/en/papers/chang-2023-speak-memory-archaeology-books-known/)
- [next → SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://md.research.elamur.ai/free/en/papers/zhang-2023-speechgpt-empowering-large-language-models/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

30 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901, 2020. [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E%2C+Mann%2C+B%2E%2C+Ryder%2C+N%2E%2C+Subbiah%2C+M%2E%2C+Kaplan%2C+J%2E+D%2E%2C+Dhariwal%2C+P%2E%2C+Neelakantan%2C+A%2E%2C+Shyam%2C+P%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+Advances+in+neural+informati)
2. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. [find](https://scholar.google.com/scholar?q=Chen%2C+M%2E%2C+Tworek%2C+J%2E%2C+Jun%2C+H%2E%2C+Yuan%2C+Q%2E%2C+Pinto%2C+H%2E+P%2E+d%2E+O%2E%2C+Kaplan%2C+J%2E%2C+Edwards%2C+H%2E%2C+Burda%2C+Y%2E%2C+Joseph%2C+N%2E%2C+Brockman%2C+G%2E%2C+et+al%2E+Evaluating+large+language+models+trained+on+code%2E+arXiv+preprint+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
3. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. [find](https://scholar.google.com/scholar?q=Chowdhery%2C+A%2E%2C+Narang%2C+S%2E%2C+Devlin%2C+J%2E%2C+Bosma%2C+M%2E%2C+Mishra%2C+G%2E%2C+Roberts%2C+A%2E%2C+Barham%2C+P%2E%2C+Chung%2C+H%2E+W%2E%2C+Sutton%2C+C%2E%2C+Gehrmann%2C+S%2E%2C+et+al%2E+Palm%3A+Scaling+language+modeling+with+pathways%2E+arXiv+preprint+arXi)
4. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. [find](https://scholar.google.com/scholar?q=Chung%2C+H%2E+W%2E%2C+Hou%2C+L%2E%2C+Longpre%2C+S%2E%2C+Zoph%2C+B%2E%2C+Tay%2C+Y%2E%2C+Fedus%2C+W%2E%2C+Li%2C+E%2E%2C+Wang%2C+X%2E%2C+Dehghani%2C+M%2E%2C+Brahma%2C+S%2E%2C+et+al%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXiv+preprint+arXiv%3A2210%2E11416%2C+202)
5. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [find](https://scholar.google.com/scholar?q=Cobbe%2C+K%2E%2C+Kosaraju%2C+V%2E%2C+Bavarian%2C+M%2E%2C+Chen%2C+M%2E%2C+Jun%2C+H%2E%2C+Kaiser%2C+L%2E%2C+Plappert%2C+M%2E%2C+Tworek%2C+J%2E%2C+Hilton%2C+J%2E%2C+Nakano%2C+R%2E%2C+et+al%2E+Training+verifiers+to+solve+math+word+problems%2E+arXiv+preprint+arXiv%3A2110) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
6. Fu, Y., Peng, H., and Khot, T. How does GPT obtain its ability? tracing emergent abilities of language models to their sources. Yao Fu’s Notion, Dec 2022. URL [https://yaofu.notion.site/](https://yaofu.notion.site/) b9a57ac0fcf74f30a1ab9e3e36fa1dc1. [↗](https://yaofu.notion.site/)
7. Ho, N., Schmid, L., and Yun, S.-Y. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071, 2022. [find](https://scholar.google.com/scholar?q=Ho%2C+N%2E%2C+Schmid%2C+L%2E%2C+and+Yun%2C+S%2E%2DY%2E+Large+language+models+are+reasoning+teachers%2E+arXiv+preprint+arXiv%3A2212%2E10071%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/ho-2023-large-language-models-are-reasoning/)
8. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. [find](https://scholar.google.com/scholar?q=Hoffmann%2C+J%2E%2C+Borgeaud%2C+S%2E%2C+Mensch%2C+A%2E%2C+Buchatskaya%2C+E%2E%2C+Cai%2C+T%2E%2C+Rutherford%2C+E%2E%2C+Casas%2C+D%2E+d%2E+L%2E%2C+Hendricks%2C+L%2E+A%2E%2C+Welbl%2C+J%2E%2C+Clark%2C+A%2E%2C+et+al%2E+Training+compute%2Doptimal+large+language+models%2E+arXiv+)
9. Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. Large language models can self-improve. arXiv preprint arXiv:2210.11610, 2022. [find](https://scholar.google.com/scholar?q=Huang%2C+J%2E%2C+Gu%2C+S%2E+S%2E%2C+Hou%2C+L%2E%2C+Wu%2C+Y%2E%2C+Wang%2C+X%2E%2C+Yu%2C+H%2E%2C+and+Han%2C+J%2E+Large+language+models+can+self%2Dimprove%2E+arXiv+preprint+arXiv%3A2210%2E11610%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/)
10. Iyer, S., Lin, X. V., Pasunuru, R., Mihaylov, T., Simig, D., Yu, P., Shuster, K., Wang, T., Liu, Q., Koura, P. S., et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.12017, 2022. [find](https://scholar.google.com/scholar?q=Iyer%2C+S%2E%2C+Lin%2C+X%2E+V%2E%2C+Pasunuru%2C+R%2E%2C+Mihaylov%2C+T%2E%2C+Simig%2C+D%2E%2C+Yu%2C+P%2E%2C+Shuster%2C+K%2E%2C+Wang%2C+T%2E%2C+Liu%2C+Q%2E%2C+Koura%2C+P%2E+S%2E%2C+et+al%2E+Opt%2Diml%3A+Scaling+language+model+instruction+meta+learning+through+the+lens+of+) [our card](https://md.research.elamur.ai/free/en/papers/iyer-2022-opt-iml-scaling-language-model/)
11. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. [find](https://scholar.google.com/scholar?q=Kaplan%2C+J%2E%2C+McCandlish%2C+S%2E%2C+Henighan%2C+T%2E%2C+Brown%2C+T%2E+B%2E%2C+Chess%2C+B%2E%2C+Child%2C+R%2E%2C+Gray%2C+S%2E%2C+Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+and+Amodei%2C+D%2E+Scaling+laws+for+neural+language+models%2E+arXiv+preprint+arXiv%3A2001%2E08361%2C+20)
12. Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stanford, CA, 2000. Morgan Kaufmann. [find](https://scholar.google.com/scholar?q=Langley%2C+P%2E+Crafting+papers+on+machine+learning%2E+In+Langley%2C+P%2E+%28ed%2E%29%2C+Proceedings+of+the+17th+International+Conference+on+Machine+Learning+%28ICML+2000%29%2C+pp%2E+1207%E2%80%931216%2C+Stanford%2C+CA%2C+2000%2E+Morgan+Kaufm)
13. Li, S., Chen, J., Shen, Y., Chen, Z., Zhang, X., Li, Z., Wang, H., Qian, J., Peng, B., Mao, Y., et al. Explanations from large language models make small reasoners better. arXiv preprint arXiv:2210.06726, 2022. [find](https://scholar.google.com/scholar?q=Li%2C+S%2E%2C+Chen%2C+J%2E%2C+Shen%2C+Y%2E%2C+Chen%2C+Z%2E%2C+Zhang%2C+X%2E%2C+Li%2C+Z%2E%2C+Wang%2C+H%2E%2C+Qian%2C+J%2E%2C+Peng%2C+B%2E%2C+Mao%2C+Y%2E%2C+et+al%2E+Explanations+from+large+language+models+make+small+reasoners+better%2E+arXiv+preprint+arXiv%3A2210%2E06)
14. Liu, L., Lewis, P., Riedel, S., and Stenetorp, P. Challenges in generalization in open domain question answering. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 2014–2029, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/20… [find](https://scholar.google.com/scholar?q=Liu%2C+L%2E%2C+Lewis%2C+P%2E%2C+Riedel%2C+S%2E%2C+and+Stenetorp%2C+P%2E+Challenges+in+generalization+in+open+domain+question+answering%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+NAACL+2022%2C+pp%2E+2014%E2%80%93202)
15. Magister, L. C., Mallinson, J., Adamek, J., Malmi, E., and Severyn, A. Teaching small language models to reason. arXiv preprint arXiv:2212.08410, 2022. [find](https://scholar.google.com/scholar?q=Magister%2C+L%2E+C%2E%2C+Mallinson%2C+J%2E%2C+Adamek%2C+J%2E%2C+Malmi%2C+E%2E%2C+and+Severyn%2C+A%2E+Teaching+small+language+models+to+reason%2E+arXiv+preprint+arXiv%3A2212%2E08410%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/magister-2023-teaching-small-language-models-reason/)
16. Min, S., Lewis, M., Zettlemoyer, L., and Hajishirzi, H. MetaICL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2791–2809, Seattle, United States, July 2022. Associat… [find](https://scholar.google.com/scholar?q=Min%2C+S%2E%2C+Lewis%2C+M%2E%2C+Zettlemoyer%2C+L%2E%2C+and+Hajishirzi%2C+H%2E+MetaICL%3A+Learning+to+learn+in+context%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+)
17. Needleman, S. B. and Wunsch, C. D. A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of molecular biology, 48(3):443–53, 1970. [find](https://scholar.google.com/scholar?q=Needleman%2C+S%2E+B%2E+and+Wunsch%2C+C%2E+D%2E+A+general+method+applicable+to+the+search+for+similarities+in+the+amino+acid+sequence+of+two+proteins%2E+Journal+of+molecular+biology%2C+48%283%29%3A443%E2%80%9353%2C+1970%2E)
18. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. [find](https://scholar.google.com/scholar?q=Ouyang%2C+L%2E%2C+Wu%2C+J%2E%2C+Jiang%2C+X%2E%2C+Almeida%2C+D%2E%2C+Wainwright%2C+C%2E+L%2E%2C+Mishkin%2C+P%2E%2C+Zhang%2C+C%2E%2C+Agarwal%2C+S%2E%2C+Slama%2C+K%2E%2C+Ray%2C+A%2E%2C+et+al%2E+Training+language+models+to+follow+instructions+with+human+feedback%2E+arXi)
19. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P. J., et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67, 2020. [find](https://scholar.google.com/scholar?q=Raffel%2C+C%2E%2C+Shazeer%2C+N%2E%2C+Roberts%2C+A%2E%2C+Lee%2C+K%2E%2C+Narang%2C+S%2E%2C+Matena%2C+M%2E%2C+Zhou%2C+Y%2E%2C+Li%2C+W%2E%2C+Liu%2C+P%2E+J%2E%2C+et+al%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext+transformer%2E+J%2E+Mach%2E+)
20. Senin, P. Dynamic time warping algorithm review. 2008. [find](https://scholar.google.com/scholar?q=Senin%2C+P%2E+Dynamic+time+warping+algorithm+review%2E+2008%2E)
21. Shridhar, K., Stolfo, A., and Sachan, M. Distilling multi-step reasoning capabilities of large language models into smaller models via semantic decompositions. arXiv preprint arXiv:2212.00193, 2022. [find](https://scholar.google.com/scholar?q=Shridhar%2C+K%2E%2C+Stolfo%2C+A%2E%2C+and+Sachan%2C+M%2E+Distilling+multi%2Dstep+reasoning+capabilities+of+large+language+models+into+smaller+models+via+semantic+decompositions%2E+arXiv+preprint+arXiv%3A2212%2E00193%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/shridhar-2023-distilling-reasoning-capabilities-into-smaller/)
22. Si, C., Gan, Z., Yang, Z., Wang, S., Wang, J., Boyd-Graber, J., and Wang, L. Prompting gpt-3 to be reliable. arXiv preprint arXiv:2210.09150, 2022. [find](https://scholar.google.com/scholar?q=Si%2C+C%2E%2C+Gan%2C+Z%2E%2C+Yang%2C+Z%2E%2C+Wang%2C+S%2E%2C+Wang%2C+J%2E%2C+Boyd%2DGraber%2C+J%2E%2C+and+Wang%2C+L%2E+Prompting+gpt%2D3+to+be+reliable%2E+arXiv+preprint+arXiv%3A2210%2E09150%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/si-2022-prompting-gpt-3-reliable/)
23. Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022. [find](https://scholar.google.com/scholar?q=Suzgun%2C+M%2E%2C+Scales%2C+N%2E%2C+Scha%CC%88rli%2C+N%2E%2C+Gehrmann%2C+S%2E%2C+Tay%2C+Y%2E%2C+Chung%2C+H%2E+W%2E%2C+Chowdhery%2C+A%2E%2C+Le%2C+Q%2E+V%2E%2C+Chi%2C+E%2E+H%2E%2C+Zhou%2C+D%2E%2C+et+al%2E+Challenging+big%2Dbench+tasks+and+whether+chain%2Dof%2Dthought+can+solve+the) [our card](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/)
24. Tan, X., Ren, Y., He, D., Qin, T., and Liu, T.-Y. Multilingual neural machine translation with knowledge distillation. In International Conference on Learning Representations, 2019. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=S1gUsoR9YX. [↗](https://openreview.net/forum?)
25. Tay, Y., Dehghani, M., Tran, V. Q., Garcia, X., Bahri, D., Schuster, T., Zheng, H. S., Houlsby, N., and Metzler, D. Unifying language learning paradigms. arXiv preprint arXiv:2205.05131, 2022. [find](https://scholar.google.com/scholar?q=Tay%2C+Y%2E%2C+Dehghani%2C+M%2E%2C+Tran%2C+V%2E+Q%2E%2C+Garcia%2C+X%2E%2C+Bahri%2C+D%2E%2C+Schuster%2C+T%2E%2C+Zheng%2C+H%2E+S%2E%2C+Houlsby%2C+N%2E%2C+and+Metzler%2C+D%2E+Unifying+language+learning+paradigms%2E+arXiv+preprint+arXiv%3A2205%2E05131%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/)
26. Thakkar, P. Copilot explorer. thakkarparth007.github.io, 2022. URL [https://thakkarparth007.github.io/copilot-explorer/posts/](https://thakkarparth007.github.io/copilot-explorer/posts/) copilot-internals.html. [↗](https://thakkarparth007.github.io/copilot-explorer/posts/)
27. Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022. [find](https://scholar.google.com/scholar?q=Thoppilan%2C+R%2E%2C+De+Freitas%2C+D%2E%2C+Hall%2C+J%2E%2C+Shazeer%2C+N%2E%2C+Kulshreshtha%2C+A%2E%2C+Cheng%2C+H%2E%2DT%2E%2C+Jin%2C+A%2E%2C+Bos%2C+T%2E%2C+Baker%2C+L%2E%2C+Du%2C+Y%2E%2C+et+al%2E+Lamda%3A+Language+models+for+dialog+applications%2E+arXiv+preprint+arXiv%3A2) [our card](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/)
28. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. [find](https://scholar.google.com/scholar?q=Wang%2C+X%2E%2C+Wei%2C+J%2E%2C+Schuurmans%2C+D%2E%2C+Le%2C+Q%2E%2C+Chi%2C+E%2E%2C+and+Zhou%2C+D%2E+Self%2Dconsistency+improves+chain+of+thought+reasoning+in+language+models%2E+arXiv+preprint+arXiv%3A2203%2E11171%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/)
29. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022a. URL https://o… [find](https://scholar.google.com/scholar?q=Wei%2C+J%2E%2C+Tay%2C+Y%2E%2C+Bommasani%2C+R%2E%2C+Raffel%2C+C%2E%2C+Zoph%2C+B%2E%2C+Borgeaud%2C+S%2E%2C+Yogatama%2C+D%2E%2C+Bosma%2C+M%2E%2C+Zhou%2C+D%2E%2C+Metzler%2C+D%2E%2C+Chi%2C+E%2E+H%2E%2C+Hashimoto%2C+T%2E%2C+Vinyals%2C+O%2E%2C+Liang%2C+P%2E%2C+Dean%2C+J%2E%2C+and+Fedus%2C+W%2E+Emergent) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/)
30. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022b. [find](https://scholar.google.com/scholar?q=Wei%2C+J%2E%2C+Wang%2C+X%2E%2C+Schuurmans%2C+D%2E%2C+Bosma%2C+M%2E%2C+Chi%2C+E%2E%2C+Le%2C+Q%2E%2C+and+Zhou%2C+D%2E+Chain+of+thought+prompting+elicits+reasoning+in+large+language+models%2E+arXiv+preprint+arXiv%3A2201%2E11903%2C+2022b%2E) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)