---
title: "Chain-of-Thought Hub — paper by Fu et al. (2023)"
description: "They introduce Chain-of-Thought Hub (CoT Hub), an open-source and continuously updated… 25 references, cited by 55 papers here."
authors: ["Yao Fu", "Litu Ou", "Mingyu Chen", "Yuhao Wan", "Hao Peng", "Tushar Khot"]
year: 2023
venue: "arXiv:2305.17306"
source: "https://arxiv.org/abs/2305.17306"
doi: "10.48550/arxiv.2305.17306"
arxiv: "2305.17306"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/fu-2023-chain-thought-hub-continuous-effort/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/fu-2023-chain-thought-hub-continuous-effort/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance

# Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance

**Authors:** [Yao Fu](https://md.research.elamur.ai/free/en/authors/yao-fu/), [Litu Ou](https://md.research.elamur.ai/free/en/authors/litu-ou/), Mingyu Chen, Yuhao Wan, [Hao Peng](https://md.research.elamur.ai/free/en/authors/hao-peng/) and [Tushar Khot](https://md.research.elamur.ai/free/en/authors/tushar-khot/) · **Year:** 2023 · **Venue:** arXiv:2305.17306 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2305.17306) · [PDF](https://arxiv.org/pdf/2305.17306) · [DOI 10.48550/arxiv.2305.17306](https://doi.org/10.48550/arxiv.2305.17306) · [arXiv 2305.17306](https://arxiv.org/abs/2305.17306) · [OpenAlex](https://openalex.org/W4378770449)

**Citations:** 14 citations (OpenAlex, 2026-10-14) · [55](https://md.research.elamur.ai/free/en/papers/fu-2023-chain-thought-hub-continuous-effort/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Chat-style comparisons make many LLMs look alike, so the authors argue that multi-step reasoning is a better way to separate strong models from weak ones. It also matters for using LLMs as engines behind real applications. They introduce Chain-of-Thought Hub (CoT Hub), an open-source and continuously updated evaluation suite. It collects six demanding benchmarks: GSM8k, MATH, MMLU, BigBench Hard, HumanEval and C-Eval. Models are tested with few-shot chain-of-thought prompting, and only final-answer accuracy is scored. Results come from the original papers where available, and the authors ran their own scripts for the rest.

Across 19 models from the GPT, Claude, PaLM, LLaMA and FlanT5 families, three patterns emerge:
- Reasoning scores rise roughly log-linearly with model size.
- Closed models (GPT-4, then claude-v1.3 and PaLM-2) clearly lead open ones.
- Most top-ranked systems have gone through RLHF.

Among open models, LLaMA 65B comes close to code-davinci-002, the base of GPT-3.5. This suggests open-source work should focus on stronger base models and RLHF.

## Key points
- The suite spans 6 benchmarks and over 100 subtasks, including Chinese via C-Eval. Code and results are at github.com/FranxYao/chain-of-thought-hub.
- Few-shot CoT prompting is chosen over zero-shot so that base and instruction-tuned checkpoints are compared fairly. This sets it apart from HeLM, which mostly uses answer-only prompts.
- Models are ranked by GSM8k. GPT-4 leads with 92.0, followed by claude-v1.3 (81.8) and PaLM-2 (80.7). LLaMA 65B reaches 50.9.
- On MMLU, LLaMA 65B scores 63.4, close to code-davinci-002 at 64.5. MATH remains very hard, with GPT-4 at only 42.5%.
- Five of the top six models are RLHF-tuned, PaLM-2 being the exception. Planned additions include commonsense, theorem-proving and tool-use datasets, plus more models such as Vicuna and Cohere.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 263 works · 180,581 citations
- [University of Edinburgh](https://md.research.elamur.ai/free/en/universities/university-of-edinburgh/) — University, Edinburgh, United Kingdom · 255 works · 68,771 citations
- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 219 works · 89,712 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng and Tushar Khot. 2023. Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance. arXiv:2305.17306.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{fu2023chain,
  title = {{Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance}},
  author = {Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng and Tushar Khot},
  year = {2023},
  doi = {10.48550/arxiv.2305.17306},
  eprint = {2305.17306},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.17306}
}
```

- [← previous Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?](https://md.research.elamur.ai/free/en/papers/hada-2023-large-language-model-based-evaluators/)
- [next → Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models](https://md.research.elamur.ai/free/en/papers/shen-2023-mixture-experts-meets-instruction-tuning/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

25 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. models to their sources. Yao Fu’s Notion, Dec 2022. URL — [PaLM 2 Technical Report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/) [https://yaofu.notion.site/](https://yaofu.notion.site/) How-does-GPT-Obtain-its-Ability-Tracing-Emergent-Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint ar… [↗](https://yaofu.notion.site/)
2. Fu, Y., Peng, H., Sabharwal, A., Clark, P., and Khot, T. [Complexity-based prompting for multi-step reasoning](https://md.research.elamur.ai/free/en/papers/fu-2022-complexity-based-prompting-multi-step/). arXiv preprint arXiv:2210.00720, 2022.
3. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. [find](https://scholar.google.com/scholar?q=Askell%2C+A%2E%2C+Bai%2C+Y%2E%2C+Chen%2C+A%2E%2C+Drain%2C+D%2E%2C+Ganguli%2C+D%2E%2C+Henighan%2C+T%2E%2C+Jones%2C+A%2E%2C+Joseph%2C+N%2E%2C+Mann%2C+B%2E%2C+DasSarma%2C+N%2E%2C+et+al%2E+A+general+language+assistant+as+a+laboratory+for+alignment%2E+arXiv+preprint+ar)
4. Geng, X., Gudibande, A., Liu, H., Wallace, E., Abbeel, P., Levine, S., and Song, D. Koala: A dialogue model for academic research. Blog post, April 2023. URL [https://bair.berkeley.edu/](https://bair.berkeley.edu/) blog/2023/04/03/koala/. [↗](https://bair.berkeley.edu/)
5. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das-Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. [find](https://scholar.google.com/scholar?q=Bai%2C+Y%2E%2C+Jones%2C+A%2E%2C+Ndousse%2C+K%2E%2C+Askell%2C+A%2E%2C+Chen%2C+A%2E%2C+Das%2DSarma%2C+N%2E%2C+Drain%2C+D%2E%2C+Fort%2C+S%2E%2C+Ganguli%2C+D%2E%2C+Henighan%2C+T%2E%2C+et+al%2E+Training+a+helpful+and+harmless+assistant+with+reinforcement+learning+from+)
6. Glaese, A., McAleese, N., Trebacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., et al. [Improving alignment of dialogue agents via targeted human judgements](https://md.research.elamur.ai/free/en/papers/glaese-2022-improving-alignment-dialogue-agents-targeted/). arXiv preprint arXiv:2209.14375, 2022.
7. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073, 2022b.
8. Gudibande, A., Wallace, E., Snell, C., Geng, X., Liu, H., Abbeel, P., Levine, S., and Song, D. [The false promise of imitating proprietary llms](https://md.research.elamur.ai/free/en/papers/gudibande-2023-false-promise-imitating-proprietary-llms/). arXiv preprint arXiv:2305.15717, 2023.
9. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021.
10. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300, 2020.
11. Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL [https://lmsys.org/blog/](https://lmsys.org/blog/) 2023-03-30-vicuna/. [↗](https://lmsys.org/blog/)
12. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). NeurIPS, 2021.
13. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311, 2022.
14. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). arXiv preprint arXiv:2203.15556, 2022.
15. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. [find](https://scholar.google.com/scholar?q=Chung%2C+H%2E+W%2E%2C+Hou%2C+L%2E%2C+Longpre%2C+S%2E%2C+Zoph%2C+B%2E%2C+Tay%2C+Y%2E%2C+Fedus%2C+W%2E%2C+Li%2C+E%2E%2C+Wang%2C+X%2E%2C+Dehghani%2C+M%2E%2C+Brahma%2C+S%2E%2C+et+al%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXiv+preprint+arXiv%3A2210%2E11416%2C+202)
16. Huang, Y., Bai, Y., Zhu, Z., Zhang, J., Zhang, J., Su, T., Liu, J., Lv, C., Zhang, Y., Lei, J., et al. [C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/). arXiv preprint arXiv:2305.08322, 2023.
17. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
18. Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). arXiv preprint arXiv:2206.14858, 2022.
19. Databricks. Free dolly: Introducing the world’s first truly open instruction-tuned llm. Blog post, April 2023. URL [https://www.databricks.com/blog/2023/04/12/](https://www.databricks.com/blog/2023/04/12/) Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., dolly-first-open-commercially-viable-instruction-tuned-llm. Yasunaga, M., Zhang,… [↗](https://www.databricks.com/blog/2023/04/12/)
20. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv:2303.08774, 2023a. arXiv preprint OpenAI. Gpt-4, 2023b. URL [https://openai.com/](https://openai.com/) research/gpt-4. [↗](https://openai.com/)
21. Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. [Challenging big-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). arXiv preprint arXiv:2210.09261, 2022.
22. Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/](https://github.com/tatsu-lab/) stanford_alpaca, 2023. [↗](https://github.com/tatsu-lab/)
23. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
24. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). arXiv preprint arXiv:2206.07682, 2022a.
25. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.[11903, 2022b](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/).

*Updated 24 October 2026*