---
title: "ThoughtSource — paper by Ott et al. (2023) — summary and ideas"
description: "Large language models often stumble on multi-step reasoning, and their decision process is hard to… 7 references, cited by 19 papers here."
authors: ["Simon Ott", "Konstantin Hebenstreit", "Valentin Lievin", "Christoffer Egeberg Hother", "Milad Moradi", "Maximilian Mayrhauser", "Robert Praas", "Ole Winther", "Matthias Samwald"]
year: 2023
venue: "arXiv:2301.11596 · Scientific Data 10, 528 (2023)"
source: "https://arxiv.org/abs/2301.11596"
doi: "10.1038/s41597-023-02433-3"
arxiv: "2301.11596"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ott-2023-thoughtsource-central-hub-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ott-2023-thoughtsource-central-hub-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / ThoughtSource: A central hub for large language model reasoning data

# ThoughtSource: A central hub for large language model reasoning data

**Authors:** [Simon Ott](https://md.research.elamur.ai/free/en/authors/simon-ott/), Konstantin Hebenstreit, Valentin Lievin, Christoffer Egeberg Hother, Milad Moradi, Maximilian Mayrhauser, Robert Praas, Ole Winther and [Matthias Samwald](https://md.research.elamur.ai/free/en/authors/matthias-samwald/) · **Year:** 2023 · **Venue:** [arXiv:2301.11596 · Scientific Data 10, 528 (2023)](https://md.research.elamur.ai/free/en/venues/scientific-data/) ([Scientific Data 2023](https://md.research.elamur.ai/free/en/venues/scientific-data/2023/)) · **Publisher:** [Nature Portfolio](https://md.research.elamur.ai/free/en/publishers/nature-portfolio/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://doi.org/10.1038/s41597-023-02433-3) · [PDF](https://arxiv.org/pdf/2301.11596) · [DOI 10.1038/s41597-023-02433-3](https://doi.org/10.1038/s41597-023-02433-3) · [arXiv 2301.11596](https://arxiv.org/abs/2301.11596) · [OpenAlex](https://openalex.org/W4385664477)

**Citations:** 49 citations (OpenAlex, 2026-10-14) · [19](https://md.research.elamur.ai/free/en/papers/ott-2023-thoughtsource-central-hub-large-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models often stumble on multi-step reasoning, and their decision process is hard to inspect. Chain-of-thought (CoT) prompting, where a model spells out intermediate steps in natural language, partly addresses this, but CoT data has been scattered across incompatible sources. ThoughtSource is an open meta-dataset and Python toolkit that brings this material together. The authors chose QA and inference benchmarks that already contained rationales or explanations. They wrote Hugging Face `datasets` loaders for each one and offer two views of every dataset: the original source schema and a unified ThoughtSource schema derived from BigBIO's QA schema. Dataset-specific converters turn expert or crowdsourced explanations into "reference CoTs" by linearizing fact lists, entailment trees and structured explanations, fixing punctuation and capitalization, and undoing tokenization. The collection also includes model-generated CoTs, some imported from earlier studies and some newly produced. Additional tools, built on LangChain, generate CoTs through LLM APIs, evaluate them, and support annotation. The resource gives researchers a common basis for analyzing, benchmarking and training reasoning in LLMs.

## Key points
- The first release covers 15 datasets: seven scientific/medical (WorldTree V2, EntailmentBank, OpenBookQA, MedQA (USMLE), MedMCQA, PubmedQA, MMLU medical subsets), three general-domain (CommonsenseQA, StrategyQA, QED) and five math word problem sets (AQUA-RAT, ASDiv, GSM8K, SVAMP, MAWPS).
- Reference CoTs exist for every dataset except PubmedQA and MedQA. Extra CoTs come from ECQA, few-shot prompting (Wei et al., Liévin et al.) and zero-shot prompting (Kojima et al.). A ThoughtSource-33 subset adds new zero-shot CoTs from several LLMs.
- A curated library of prompt fragments (instructions, CoT triggers such as "Let's think step by step", answer-extraction templates) makes it possible to test systematically how prompt wording affects results.
- Two web tools are included: a Dataset Viewer for browsing and an Annotator that highlights similar text across CoTs and records judgments on individual chains.
- Code and generated data are MIT-licensed and hosted at github.com/OpenBioLink/ThoughtSource, with a Zenodo archive. Adapted data keeps the licenses of its original sources.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [KTH Royal Institute of Technology](https://md.research.elamur.ai/free/en/universities/kth-royal-institute-of-technology/) — University, Stockholm, Sweden · 18 works · 788 citations
- [Medical University of Vienna](https://md.research.elamur.ai/free/en/universities/medical-university-of-vienna/) — University, Vienna, Austria · 3 works · 161 citations
- [Technical University of Denmark](https://md.research.elamur.ai/free/en/universities/technical-university-of-denmark/) — University, Kongens Lyngby, Denmark · 4 works · 698 citations
- [Copenhagen University Hospital](https://md.research.elamur.ai/free/en/hospitals/copenhagen-university-hospital/) — Hospital, Copenhagen, Denmark · 4 works · 1,008 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Simon Ott, Konstantin Hebenstreit, Valentin Lievin, Christoffer Egeberg Hother, Milad Moradi, Maximilian Mayrhauser, Robert Praas, Ole Winther and Matthias Samwald. 2023. ThoughtSource: A central hub for large language model reasoning data. arXiv:2301.11596 · Scientific Data 10, 528 (2023).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ott2023thoughtsource,
  title = {{ThoughtSource: A central hub for large language model reasoning data}},
  author = {Simon Ott, Konstantin Hebenstreit, Valentin Lievin, Christoffer Egeberg Hother, Milad Moradi, Maximilian Mayrhauser, Robert Praas, Ole Winther and Matthias Samwald},
  year = {2023},
  doi = {10.1038/s41597-023-02433-3},
  eprint = {2301.11596},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1038/s41597-023-02433-3}
}
```

- [← previous Preventing Generation of Verbatim Memorization in Language Models Gives a False Sense of Privacy](https://md.research.elamur.ai/free/en/papers/ippolito-2023-preventing-generation-verbatim-memorization-language/)
- [next → ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Text](https://md.research.elamur.ai/free/en/papers/hasanain-2023-araieval-shared-task-persuasion-techniques/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

7 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 1. Sanh, V. et al. [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/). Preprint at https://doi.org/10.48550/arXiv.2110.08207 (2021). \#\# 2. Brown, T. B. et al. Language Models are Few-Shot Learners. Preprint at https://doi.org/10.48550/arXiv.2005.14165 (2020). \#\# 3. Ouyang, L. et al. T… [↗](https://doi.org/10.48550/arXiv.2110.08207)
2. Preprint at https://doi.org/10.48550/arxiv.2203.[02155 (2022)](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). 4. OpenAI. GPT-4 Technical Report. Preprint at https://doi.org/10.48550/arXiv.2303.08774 (2023). \#\# 5. Wei, J. et al. Chain of Thought Prompting Elicits Reasoning in Large Language Models. [↗](https://doi.org/10.48550/arxiv.2203.02155)
3. Preprint at https://doi.org/10.48550/arxiv.2201.[11903 (2022)](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). 6. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y. & Iwasawa, Y. Large Language Models are Zero-Shot Reasoners. Preprint at https://doi.org/10.48550/arxiv.2205.11916 (2022). 7. Huang, J. & Chang, K. C.-C. Towards Reasoning in Large Language M… [↗](https://doi.org/10.48550/arxiv.2201.11903)
4. Preprint at https://doi.org/10.48550/arxiv.2210.03350 (2022). 11.Jung, J. et al. [Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations](https://md.research.elamur.ai/free/en/papers/jung-2022-maieutic-prompting-logically-consistent-reasoning/). Preprint at https://doi.org/10.48550/arxiv.2205.11822 (2022). 12.Dua, D., Gupta, S., Singh, S. & Gardner, M. Successive Prompting for Decom… [↗](https://doi.org/10.48550/arxiv.2210.03350)
5. Trans. Assoc. Comput. Linguist. 9, 790–806 (2021). 33.Ling, W., Yogatama, D., Dyer, C. & Blunsom, P. Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems. in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1:… [find](https://scholar.google.com/scholar?q=Trans%2E+Assoc%2E+Comput%2E+Linguist%2E+9%2C+790%E2%80%93806+%282021%29%2E+33%2ELing%2C+W%2E%2C+Yogatama%2C+D%2E%2C+Dyer%2C+C%2E+%26+Blunsom%2C+P%2E+Program+Induction+by+Rationale+Generation%3A+Learning+to+Solve+and+Explain+Algebraic+Word+Problems%2E+i)
6. (Association for Computational Linguistics, 2017). doi:10.18653/v1/P17-1015. 34.Miao, S., Liang, C.-C. & Su, K.-Y. A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers. in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics 975–984 (Ass… [find](https://scholar.google.com/scholar?q=%28Association+for+Computational+Linguistics%2C+2017%29%2E+doi%3A10%2E18653%2Fv1%2FP17%2D1015%2E+34%2EMiao%2C+S%2E%2C+Liang%2C+C%2E%2DC%2E+%26+Su%2C+K%2E%2DY%2E+A+Diverse+Corpus+for+Evaluating+and+Developing+English+Math+Word+Problem+Solvers%2E+in+)
7. (Association for Computational Linguistics, 2016). doi:10.18653/v1/N16-1136. 37.Patel, A., Bhattamishra, S. & Goyal, N. [Are NLP Models really able to Solve Simple Math Word Problems?](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/) in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics:…

*Updated 30 October 2026*