---
title: "Premise Order Matters in Reasoning with Large Language Models"
description: "Changing the order of premises does not change a logical or arithmetic problem, but this paper shows… 38 references, cited by 55 papers here."
authors: ["Xinyun Chen", "Ryan A. Chi", "Xuezhi Wang", "Denny Zhou"]
year: 2024
venue: "arXiv:2402.08939"
source: "https://arxiv.org/abs/2402.08939"
doi: "10.48550/arxiv.2402.08939"
arxiv: "2402.08939"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2024-premise-order-matters-reasoning-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2024-premise-order-matters-reasoning-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / Premise Order Matters in Reasoning with Large Language Models

# Premise Order Matters in Reasoning with Large Language Models

**Authors:** [Xinyun Chen](https://md.research.elamur.ai/free/en/authors/xinyun-chen/), Ryan A. Chi, [Xuezhi Wang](https://md.research.elamur.ai/free/en/authors/xuezhi-wang/) and [Denny Zhou](https://md.research.elamur.ai/free/en/authors/denny-zhou/) · **Year:** 2024 · **Venue:** arXiv:2402.08939 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.08939) · [PDF](https://arxiv.org/pdf/2402.08939) · [DOI 10.48550/arxiv.2402.08939](https://doi.org/10.48550/arxiv.2402.08939) · [arXiv 2402.08939](https://arxiv.org/abs/2402.08939) · [OpenAlex](https://openalex.org/W4391835676)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [55](https://md.research.elamur.ai/free/en/papers/chen-2024-premise-order-matters-reasoning-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Changing the order of premises does not change a logical or arithmetic problem, but this paper shows that large language models are highly sensitive to it. The authors built two test resources. The first is a synthetic deductive-reasoning benchmark adapted from SimpleLogic. It contains modus ponens problems whose premises are shuffled to set distances from the order of the ground-truth proof, measured with Kendall tau. The second is R-GSM, in which GSM8K word problems have their sentences reordered by hand without altering the answer. They evaluated GPT-4-turbo, GPT-3.5-turbo, PaLM 2-L and Gemini 1.0 Pro using zero-shot greedy decoding. Every model did best when premises appeared in the same sequence as the proof steps. Accuracy could fall by more than 30% under other orderings, and irrelevant rules made the effect worse. The finding suggests that LLMs prefer to reason in a single left-to-right pass rather than jumping around the input. This is a robustness issue that matters for any prompt-based reasoning pipeline.

## Key points
- **Logic benchmark:** proofs need 4–12 rules, with 0, 5 or 10 distracting rules added. There are 200 problems per rule count and 15 order or distractor variants each, giving about 27K items. A proof counts as correct only if every step is valid.
- **Order variants:** premise orders are labelled by normalized Kendall tau, from forward (τ = 1) through 0.5, 0 and −0.5 to backward (τ = −1).
- **Model differences:**
  - The GPT models handle the reversed (backward-chaining) order better than random shuffles.
  - PaLM 2-L tends to perform worst on the reversed order.
- **R-GSM:** 220 pairs of original and reordered GSM8K test problems, each with 5–8 sentences. All models lose accuracy on the reordered versions, especially on longer problems.
- **Ruling out context length:** inputs are under 300 tokens, so the lost-in-the-middle effect does not explain the results. The authors leave fixes for the order effect to future work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 567 works · 335,189 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,948 works

## Cite

Cite as: Xinyun Chen, Ryan A. Chi, Xuezhi Wang and Denny Zhou. 2024. Premise Order Matters in Reasoning with Large Language Models. arXiv:2402.08939.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chen2024premise,
  title = {{Premise Order Matters in Reasoning with Large Language Models}},
  author = {Xinyun Chen, Ryan A. Chi, Xuezhi Wang and Denny Zhou},
  year = {2024},
  doi = {10.48550/arxiv.2402.08939},
  eprint = {2402.08939},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.08939}
}
```

- [← previous Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process](https://md.research.elamur.ai/free/en/papers/ye-2024-physics-language-models-part-2/)
- [next → A New Massive Multilingual Dataset for High-Performance Language Technologies](https://md.research.elamur.ai/free/en/papers/gibert-2024-new-massive-multilingual-dataset-high/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

38 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. Abdou, V. Ravishankar, A. Kulmizev, and A. Søgaard. Word order does matter and shuffled language models know it. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6907–6919, 2022. [find](https://scholar.google.com/scholar?q=M%2E+Abdou%2C+V%2E+Ravishankar%2C+A%2E+Kulmizev%2C+and+A%2E+S%C3%B8gaard%2E+Word+order+does+matter+and+shuffled+language+models+know+it%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Lingu)
2. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732, 2021.
3. L. Berglund, M. Tong, M. Kaufmann, M. Balesni, A. C. Stickland, T. Korbak, and O. Evans. The reversal curse: Llms trained on" a is b" fail to learn" b is a". arXiv preprint arXiv:2309.12288, 2023. [find](https://scholar.google.com/scholar?q=L%2E+Berglund%2C+M%2E+Tong%2C+M%2E+Kaufmann%2C+M%2E+Balesni%2C+A%2E+C%2E+Stickland%2C+T%2E+Korbak%2C+and+O%2E+Evans%2E+The+reversal+curse%3A+Llms+trained+on%22+a+is+b%22+fail+to+learn%22+b+is+a%22%2E+arXiv+preprint+arXiv%3A2309%2E12288%2C+2023%2E)
4. S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712, 2023.
5. Q. Cao, T. Kojima, Y. Matsuo, and Y. Iwasawa. Unnatural error correction: Gpt-4 can almost perfectly handle unnatural scrambled text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8898–8913, 2023. [find](https://scholar.google.com/scholar?q=Q%2E+Cao%2C+T%2E+Kojima%2C+Y%2E+Matsuo%2C+and+Y%2E+Iwasawa%2E+Unnatural+error+correction%3A+Gpt%2D4+can+almost+perfectly+handle+unnatural+scrambled+text%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natu)
6. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021.
7. V. A. Cicirello. Kendall tau sequence distance: Extending kendall tau from ranks to sequences. arXiv preprint arXiv:1905.02752, 2019. — [arXiv:1905.02752](https://arxiv.org/abs/1905.02752) [find](https://scholar.google.com/scholar?q=V%2E+A%2E+Cicirello%2E+Kendall+tau+sequence+distance%3A+Extending+kendall+tau+from+ranks+to+sequences%2E+arXiv+preprint+arXiv%3A1905%2E02752%2C+2019%2E+%E2%80%94+%5BarXiv%3A1905%2E02752%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1905%2E02752%29)
8. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
9. M. Dekeyser, W. Schroyens, W. Schaeken, O. Spitaels, and G. d’Ydewalle. Preferred premise order in propositional reasoning: Semantic informativeness and co-reference. Deductive reasoning and strategies, pages 73–95, 2000. [find](https://scholar.google.com/scholar?q=M%2E+Dekeyser%2C+W%2E+Schroyens%2C+W%2E+Schaeken%2C+O%2E+Spitaels%2C+and+G%2E+d%E2%80%99Ydewalle%2E+Preferred+premise+order+in+propositional+reasoning%3A+Semantic+informativeness+and+co%2Dreference%2E+Deductive+reasoning+and+strategie)
10. D. Ferreira and A. Freitas. [Premise selection in natural language mathematical texts](https://md.research.elamur.ai/free/en/papers/ferreira-2020-premise-selection-natural-language-mathematical/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7365–7374, 2020.
11. Gemini. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
12. V. Girotto, A. Mazzocco, and A. Tasso. The effect of premise order in conditional reasoning: A test of the mental model theory. Cognition, 63(1):1–28, 1997. [find](https://scholar.google.com/scholar?q=V%2E+Girotto%2C+A%2E+Mazzocco%2C+and+A%2E+Tasso%2E+The+effect+of+premise+order+in+conditional+reasoning%3A+A+test+of+the+mental+model+theory%2E+Cognition%2C+63%281%29%3A1%E2%80%9328%2C+1997%2E)
13. Google. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403, 2023.
14. T. Hagendorff, S. Fabi, and M. Kosinski. Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in chatgpt. Nature Computational Science, 3(10):833–838, 2023. [find](https://scholar.google.com/scholar?q=T%2E+Hagendorff%2C+S%2E+Fabi%2C+and+M%2E+Kosinski%2E+Human%2Dlike+intuitive+behavior+and+reasoning+biases+emerged+in+large+language+models+but+disappeared+in+chatgpt%2E+Nature+Computational+Science%2C+3%2810%29%3A833%E2%80%93838%2C+20)
15. S. Han, H. Schoelkopf, Y. Zhao, Z. Qi, M. Riddell, L. Benson, L. Sun, E. Zubova, Y. Qiao, M. Burtell, et al. [Folio: Natural language reasoning with first-order logic](https://md.research.elamur.ai/free/en/papers/han-2022-folio-natural-language-reasoning-first/). arXiv preprint arXiv:2209.00840, 2022.
16. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874, 2021.
17. G. Irving, C. Szegedy, A. A. Alemi, N. Eén, F. Chollet, and J. Urban. Deepmath-deep sequence models for premise selection. Advances in neural information processing systems, 29, 2016. — [DeepMath - Deep Sequence Models for Premise Selection](https://md.research.elamur.ai/free/en/papers/alemi-2016-deepmath-deep-sequence-models-premise/)
18. E. Jones and J. Steinhardt. Capturing failures of large language models via human cognitive biases. Advances in Neural Information Processing Systems, 35:11785–11799, 2022. [find](https://scholar.google.com/scholar?q=E%2E+Jones+and+J%2E+Steinhardt%2E+Capturing+failures+of+large+language+models+via+human+cognitive+biases%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A11785%E2%80%9311799%2C+2022%2E)
19. N. Lee, K. Sreenivasan, J. D. Lee, K. Lee, and D. Papailiopoulos. [Teaching arithmetic to small transformers](https://md.research.elamur.ai/free/en/papers/lee-2023-teaching-arithmetic-small-transformers/). arXiv preprint arXiv:2307.03381, 2023.
20. Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al. [Competition-level code generation with alphacode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/). Science, 378(6624):1092– 1097, 2022.
21. N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). Transactions of the Association for Computational Linguistics, 12:157–173, 2024.
22. R. T. McCoy, S. Yao, D. Friedman, M. Hardy, and T. L. Griffiths. Embers of autoregression: Understanding large language models through the problem they are trained to solve. arXiv preprint arXiv:2309.13638, 2023. [find](https://scholar.google.com/scholar?q=R%2E+T%2E+McCoy%2C+S%2E+Yao%2C+D%2E+Friedman%2C+M%2E+Hardy%2C+and+T%2E+L%2E+Griffiths%2E+Embers+of+autoregression%3A+Understanding+large+language+models+through+the+problem+they+are+trained+to+solve%2E+arXiv+preprint+arXiv%3A2309%2E)
23. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
24. A. Saparov and H. He. [Language models are greedy reasoners: A systematic formal analysis of chain-of-thought](https://md.research.elamur.ai/free/en/papers/saparov-2022-language-models-greedy-reasoners-systematic/). arXiv preprint arXiv:2210.01240, 2022.
25. A. Saparov, R. Y. Pang, V. Padmakumar, N. Joshi, S. M. Kazemi, N. Kim, and H. He. [Testing the general deductive reasoning capacity of large language models using ood examples](https://md.research.elamur.ai/free/en/papers/saparov-2023-testing-general-deductive-reasoning-capacity/). arXiv preprint arXiv:2305.15269, 2023.
26. P. K. Sen. Estimates of the regression coefficient based on kendall’s tau. Journal of the American statistical association, 63(324):1379–1389, 1968. [find](https://scholar.google.com/scholar?q=P%2E+K%2E+Sen%2E+Estimates+of+the+regression+coefficient+based+on+kendall%E2%80%99s+tau%2E+Journal+of+the+American+statistical+association%2C+63%28324%29%3A1379%E2%80%931389%2C+1968%2E)
27. F. Shi, X. Chen, K. Misra, N. Scales, D. Dohan, E. H. Chi, N. Schärli, and D. Zhou. [Large language models can be easily distracted by irrelevant context](https://md.research.elamur.ai/free/en/papers/shi-2023-large-language-models-easily-distracted/). In International Conference on Machine Learning, pages 31210–31227. PMLR, 2023.
28. K. Sinha, P. Parthasarathi, J. Pineau, and A. Williams. [Unnatural language inference](https://md.research.elamur.ai/free/en/papers/sinha-2021-unnatural-language-inference/). arXiv preprint arXiv:2101.00010, 2020.
29. Y. Wan, W. Wang, Y. Yang, Y. Yuan, J.-t. Huang, P. He, W. Jiao, and M. R. Lyu. A & b== b & a: Triggering logical reasoning failures in large language models. arXiv preprint arXiv:2401.00757, 2024. — [arXiv:2401.00757](https://arxiv.org/abs/2401.00757) [find](https://scholar.google.com/scholar?q=Y%2E+Wan%2C+W%2E+Wang%2C+Y%2E+Yang%2C+Y%2E+Yuan%2C+J%2E%2Dt%2E+Huang%2C+P%2E+He%2C+W%2E+Jiao%2C+and+M%2E+R%2E+Lyu%2E+A+%26+b%3D%3D+b+%26+a%3A+Triggering+logical+reasoning+failures+in+large+language+models%2E+arXiv+preprint+arXiv%3A2401%2E00757%2C+2024%2E+%E2%80%94+%5B)
30. M. Wang, Y. Tang, J. Wang, and J. Deng. [Premise selection for theorem proving by deep graph embedding](https://md.research.elamur.ai/free/en/papers/wang-2017-premise-selection-theorem-proving-deep/). Advances in neural information processing systems, 30, 2017.
31. P. Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y. Cao, Q. Liu, T. Liu, and Z. Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023. [find](https://scholar.google.com/scholar?q=P%2E+Wang%2C+L%2E+Li%2C+L%2E+Chen%2C+Z%2E+Cai%2C+D%2E+Zhu%2C+B%2E+Lin%2C+Y%2E+Cao%2C+Q%2E+Liu%2C+T%2E+Liu%2C+and+Z%2E+Sui%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv+preprint+arXiv%3A2305%2E17926%2C+2023%2E)
32. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
33. F. Xu, Q. Lin, J. Han, T. Zhao, J. Liu, and E. Cambria. Are large language models really good logical reasoners? a comprehensive evaluation from deductive, inductive and abductive views. arXiv preprint arXiv:2306.[09841, 2023](https://md.research.elamur.ai/free/en/papers/xu-2023-large-language-models-really-good/).
34. S. Yan, C. Shen, J. Liu, and J. Ye. Concise and organized perception facilitates large language models for deductive reasoning. arXiv preprint arXiv:2310.03309, 2023. — [arXiv:2310.03309](https://arxiv.org/abs/2310.03309) [find](https://scholar.google.com/scholar?q=S%2E+Yan%2C+C%2E+Shen%2C+J%2E+Liu%2C+and+J%2E+Ye%2E+Concise+and+organized+perception+facilitates+large+language+models+for+deductive+reasoning%2E+arXiv+preprint+arXiv%3A2310%2E03309%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E03309%5D%28https%3A%2F%2Farxi)
35. H. Zhang, L. H. Li, T. Meng, K.-W. Chang, and G. V. d. Broeck. On the paradox of learning to reason from data. arXiv preprint arXiv:2205.11502, 2022. — [arXiv:2205.11502](https://arxiv.org/abs/2205.11502) [find](https://scholar.google.com/scholar?q=H%2E+Zhang%2C+L%2E+H%2E+Li%2C+T%2E+Meng%2C+K%2E%2DW%2E+Chang%2C+and+G%2E+V%2E+d%2E+Broeck%2E+On+the+paradox+of+learning+to+reason+from+data%2E+arXiv+preprint+arXiv%3A2205%2E11502%2C+2022%2E+%E2%80%94+%5BarXiv%3A2205%2E11502%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2205%2E11)
36. H. Zhou, A. Bradley, E. Littwin, N. Razin, O. Saremi, J. Susskind, S. Bengio, and P. Nakkiran. [What algorithms can transformers learn? a study in length generalization](https://md.research.elamur.ai/free/en/papers/zhou-2023-algorithms-transformers-learn-study-length/). arXiv preprint arXiv:2310.16028, 2023.
37. Y. Zhou, U. Alon, X. Chen, X. Wang, R. Agarwal, and D. Zhou. [Transformers can achieve length generalization but not robustly](https://md.research.elamur.ai/free/en/papers/zhou-2024-transformers-achieve-length-generalization-but/). arXiv preprint arXiv:2402.09371, 2024.
38. Z. Zhu, Y. Xue, X. Chen, D. Zhou, J. Tang, D. Schuurmans, and H. Dai. Large language models can learn rules. arXiv preprint arXiv:2310.07064, 2023. — [arXiv:2310.07064](https://arxiv.org/abs/2310.07064) [find](https://scholar.google.com/scholar?q=Z%2E+Zhu%2C+Y%2E+Xue%2C+X%2E+Chen%2C+D%2E+Zhou%2C+J%2E+Tang%2C+D%2E+Schuurmans%2C+and+H%2E+Dai%2E+Large+language+models+can+learn+rules%2E+arXiv+preprint+arXiv%3A2310%2E07064%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E07064%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E0706)

*Updated 3 November 2026*