---
title: "Evaluating Large Language Models with NeuBAROCO:"
description: "The authors build NeuBAROCO, a bilingual English–Japanese resource derived from BAROCO, a… 27 references, cited by 15 papers here."
authors: ["Risako Ando", "Takanobu Morishita", "Hirohiko Abe", "Koji Mineshima", "Mitsuhiro Okada"]
year: 2023
venue: "arXiv:2306.12567"
source: "https://arxiv.org/abs/2306.12567"
doi: "10.48550/arxiv.2306.12567"
arxiv: "2306.12567"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ando-2023-evaluating-large-language-models-neubaroco/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ando-2023-evaluating-large-language-models-neubaroco/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/) / Evaluating Large Language Models with NeuBAROCO: Syllogistic Reasoning Ability and Human-like Biases

# Evaluating Large Language Models with NeuBAROCO: Syllogistic Reasoning Ability and Human-like Biases

**Authors:** Risako Ando, Takanobu Morishita, Hirohiko Abe, [Koji Mineshima](https://md.research.elamur.ai/free/en/authors/koji-mineshima/) and Mitsuhiro Okada · **Year:** 2023 · **Venue:** arXiv:2306.12567 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2306.12567) · [PDF](https://arxiv.org/pdf/2306.12567) · [DOI 10.48550/arxiv.2306.12567](https://doi.org/10.48550/arxiv.2306.12567) · [arXiv 2306.12567](https://arxiv.org/abs/2306.12567) · [OpenAlex](https://openalex.org/W4381826854)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [15](https://md.research.elamur.ai/free/en/papers/ando-2023-evaluating-large-language-models-neubaroco/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether large language models fall into the same reasoning traps that psychologists have documented in people solving syllogisms. The authors build NeuBAROCO, a bilingual English–Japanese resource derived from BAROCO, a question collection previously used to measure human deductive skill, including in twin studies. Each original multiple-choice item was recast in natural language inference format: two premises plus one candidate conclusion, hand-labelled as entailment, contradiction, or neutral. Items are also grouped as symbolic, belief-consistent, or belief-inconsistent, and problems where humans typically slip through conversion errors or atmosphere effects are marked. RoBERTa and BART (English) and GPT-3.5 (English and Japanese) were tested without any solved examples. No model proved dependable. GPT-3.5 answered only about half of the problems correctly. Every system scored lowest on belief-inconsistent items and on conversion and atmosphere cases, where RoBERTa and BART were close to zero. The work connects cognitive-science research on reasoning biases with LLM evaluation. It also shows that the bias pattern appears in two typologically distant languages.

## Key points
- NeuBAROCO has 375 inference problems: 122 entailment, 71 contradiction and 182 neutral. They were derived from BAROCO-ALL, a 200-question subset of BAROCO (Shikishima et al.).
- The evaluation covers three human-like biases: belief bias, conversion errors and atmosphere effects. Problems are also split by content type (symbol / consistent / inconsistent).
- Overall accuracy was:
  - RoBERTa: 34.67%
  - BART: 35.20%
  - GPT-3.5 (English): 51.73%
  - GPT-3.5 (Japanese): 48.27%
- English GPT-3.5 handled entailment well (79.51%) but struggled with contradiction (38.03%) and neutral items (38.46%).
- On bias-prone items, accuracy dropped sharply:
  - Conversion: 0% for RoBERTa, 1.43% for BART, 25.71% for GPT-3.5 (EN) and 21.43% for GPT-3.5 (JA).
  - Atmosphere: 0% for RoBERTa, 0.96% for BART, 39.42% for GPT-3.5 (EN) and 22.12% for GPT-3.5 (JA).
  - Belief-inconsistent: the weakest content category for every model, with GPT-3.5 (EN) at 31.37%.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Keio University](https://md.research.elamur.ai/free/en/universities/keio-university/) — University, Tokyo, Japan · 7 works · 882 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Risako Ando, Takanobu Morishita, Hirohiko Abe, Koji Mineshima and Mitsuhiro Okada. 2023. Evaluating Large Language Models with NeuBAROCO: Syllogistic Reasoning Ability and Human-like Biases. arXiv:2306.12567.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ando2023evaluating,
  title = {{Evaluating Large Language Models with NeuBAROCO: Syllogistic Reasoning Ability and Human-like Biases}},
  author = {Risako Ando, Takanobu Morishita, Hirohiko Abe, Koji Mineshima and Mitsuhiro Okada},
  year = {2023},
  doi = {10.48550/arxiv.2306.12567},
  eprint = {2306.12567},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.12567}
}
```

- [← previous Crossing the Threshold: Idiomatic Machine Translation through Retrieval Augmentation and Loss Weighting](https://md.research.elamur.ai/free/en/papers/liu-2023-crossing-threshold-idiomatic-machine-translation/)
- [next → Guiding Computational Stance Detection with Expanded Stance Triangle Framework](https://md.research.elamur.ai/free/en/papers/liu-2023-guiding-computational-stance-detection-expanded/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/)

## References

27 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
2. Nick Chater and Mike Oaksford. 1999. The probability heuristics model of syllogistic reasoning. Cognitive Psychology, 38(2):191–258. [find](https://scholar.google.com/scholar?q=Nick+Chater+and+Mike+Oaksford%2E+1999%2E+The+probability+heuristics+model+of+syllogistic+reasoning%2E+Cognitive+Psychology%2C+38%282%29%3A191%E2%80%93258%2E)
3. Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. 2022. Language models show human-like content effects on reasoning. arXiv preprint arXiv:2207.07051. [find](https://scholar.google.com/scholar?q=Ishita+Dasgupta%2C+Andrew+K+Lampinen%2C+Stephanie+CY+Chan%2C+Antonia+Creswell%2C+Dharshan+Kumaran%2C+James+L+McClelland%2C+and+Felix+Hill%2E+2022%2E+Language+models+show+human%2Dlike+content+effects+on+reasoning%2E+arXiv)
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
5. Jonathan St.B. T. Evans. 1989. Bias in Human Reasoning: Causes and Consequences. Lawrence Erlbaum Associates, Inc. [find](https://scholar.google.com/scholar?q=Jonathan+St%2EB%2E+T%2E+Evans%2E+1989%2E+Bias+in+Human+Reasoning%3A+Causes+and+Consequences%2E+Lawrence+Erlbaum+Associates%2C+Inc%2E)
6. Jonathan St.B. T. Evans, Stephen E. Newstead, and Ruth M. J. Byrne. 1993. Human Reasoning: The Psychology of Deduction. Psychology Press. [find](https://scholar.google.com/scholar?q=Jonathan+St%2EB%2E+T%2E+Evans%2C+Stephen+E%2E+Newstead%2C+and+Ruth+M%2E+J%2E+Byrne%2E+1993%2E+Human+Reasoning%3A+The+Psychology+of+Deduction%2E+Psychology+Press%2E)
7. Bart Geurts. 2003. Reasoning with quantifiers. Cognition, 86(3):223–251. [find](https://scholar.google.com/scholar?q=Bart+Geurts%2E+2003%2E+Reasoning+with+quantifiers%2E+Cognition%2C+86%283%29%3A223%E2%80%93251%2E)
8. Vinod Goel, Christian Buchel, Chris Frith, and Raymond J Dolan. 2000. Dissociation of mechanisms underlying syllogistic reasoning. Neuroimage, 12(5):504–514. [find](https://scholar.google.com/scholar?q=Vinod+Goel%2C+Christian+Buchel%2C+Chris+Frith%2C+and+Raymond+J+Dolan%2E+2000%2E+Dissociation+of+mechanisms+underlying+syllogistic+reasoning%2E+Neuroimage%2C+12%285%29%3A504%E2%80%93514%2E)
9. Sangeet Khemlani and Philip N Johnson-Laird. 2012. Theories of the syllogism: A meta-analysis. Psychological Bulletin, 138(3):427. [find](https://scholar.google.com/scholar?q=Sangeet+Khemlani+and+Philip+N+Johnson%2DLaird%2E+2012%2E+Theories+of+the+syllogism%3A+A+meta%2Danalysis%2E+Psychological+Bulletin%2C+138%283%29%3A427%2E)
10. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the…
11. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [RoBERTa: A robustly optimized BERT pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692.
12. Lawrence S. Moss. 2015. Natural logic. In Shalom Lappin and Chris Fox, editors, The Handbook of Contemporary Semantic Theory, 2 edition, pages 559– 592. Wiley. [find](https://scholar.google.com/scholar?q=Lawrence+S%2E+Moss%2E+2015%2E+Natural+logic%2E+In+Shalom+Lappin+and+Chris+Fox%2C+editors%2C+The+Handbook+of+Contemporary+Semantic+Theory%2C+2+edition%2C+pages+559%E2%80%93+592%2E+Wiley%2E)
13. Stephen E. Newstead. 1989. Interpretational errors in syllogistic reasoning. Journal of Memory and Language, 28(1):78–91. [find](https://scholar.google.com/scholar?q=Stephen+E%2E+Newstead%2E+1989%2E+Interpretational+errors+in+syllogistic+reasoning%2E+Journal+of+Memory+and+Language%2C+28%281%29%3A78%E2%80%9391%2E)
14. Stephen E Newstead, Paul Pollard, Jonathan St BT Evans, and Julie L Allen. 1992. The source of belief bias effects in syllogistic reasoning. Cognition, 45(3):257–284. [find](https://scholar.google.com/scholar?q=Stephen+E+Newstead%2C+Paul+Pollard%2C+Jonathan+St+BT+Evans%2C+and+Julie+L+Allen%2E+1992%2E+The+source+of+belief+bias+effects+in+syllogistic+reasoning%2E+Cognition%2C+45%283%29%3A257%E2%80%93284%2E)
15. Kyle Richardson, Hai Hu, Lawrence Moss, and Ashish Sabharwal. 2020. [Probing natural language inference models through semantic fragments](https://md.research.elamur.ai/free/en/papers/richardson-2019-probing-natural-language-inference-models/). In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8713–8721.
16. Víctor Sánchez Valencia. 1991. Studies on Natural Logic and Categorial Grammar. Ph.D. thesis, University of Amsterdam. [find](https://scholar.google.com/scholar?q=V%C3%ADctor+S%C3%A1nchez+Valencia%2E+1991%2E+Studies+on+Natural+Logic+and+Categorial+Grammar%2E+Ph%2ED%2E+thesis%2C+University+of+Amsterdam%2E)
17. Yuri Sato and Koji Mineshima. 2015. How diagrams can support syllogistic reasoning: an experimental study. Journal of Logic, Language and Information, 24:409–455. [find](https://scholar.google.com/scholar?q=Yuri+Sato+and+Koji+Mineshima%2E+2015%2E+How+diagrams+can+support+syllogistic+reasoning%3A+an+experimental+study%2E+Journal+of+Logic%2C+Language+and+Information%2C+24%3A409%E2%80%93455%2E)
18. Viktor Schlegel, Kamen Pavlov, and Ian Pratt-Hartmann. 2022. Can transformers reason in fragments of natural language? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11184–11199, Abu Dhabi, United Arab Emirates. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Viktor+Schlegel%2C+Kamen+Pavlov%2C+and+Ian+Pratt%2DHartmann%2E+2022%2E+Can+transformers+reason+in+fragments+of+natural+language%3F+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Language+Pr)
19. Chizuru Shikishima, Juko Ando, Pierre Grialou, Ryo Takemura, and Mitsuhiro Okada. 2006. A behavioral genetic study of syllogism solving using linguistic and graphical representations: a preliminary report. In Images and Reasoning, pages 69–85. Keio University Press. [find](https://scholar.google.com/scholar?q=Chizuru+Shikishima%2C+Juko+Ando%2C+Pierre+Grialou%2C+Ryo+Takemura%2C+and+Mitsuhiro+Okada%2E+2006%2E+A+behavioral+genetic+study+of+syllogism+solving+using+linguistic+and+graphical+representations%3A+a+preliminary+re)
20. Chizuru Shikishima, Kai Hiraishi, Shinji Yamagata, Juko Ando, and Mitsuhiro Okada. 2015. Genetic factors of individual differences in decision making in economic behavior: A Japanese twin study using the Allais problem. Frontiers in Psychology, 6:1712. [find](https://scholar.google.com/scholar?q=Chizuru+Shikishima%2C+Kai+Hiraishi%2C+Shinji+Yamagata%2C+Juko+Ando%2C+and+Mitsuhiro+Okada%2E+2015%2E+Genetic+factors+of+individual+differences+in+decision+making+in+economic+behavior%3A+A+Japanese+twin+study+using+)
21. Chizuru Shikishima, Kai Hiraishi, Shinji Yamagata, Yutaro Sugimoto, Ryo Takemura, Koken Ozaki, Mitsuhiro Okada, Tatsushi Toda, and Juko Ando. 2009. Is g an entity? a Japanese twin study using syllogisms and intelligence tests. Intelligence, 37(3):256–267. [find](https://scholar.google.com/scholar?q=Chizuru+Shikishima%2C+Kai+Hiraishi%2C+Shinji+Yamagata%2C+Yutaro+Sugimoto%2C+Ryo+Takemura%2C+Koken+Ozaki%2C+Mitsuhiro+Okada%2C+Tatsushi+Toda%2C+and+Juko+Ando%2E+2009%2E+Is+g+an+entity%3F+a+Japanese+twin+study+using+syllogis)
22. Keith Stenning and Michiel Van Lambalgen. 2012. Human Reasoning and Cognitive Science. MIT Press. [find](https://scholar.google.com/scholar?q=Keith+Stenning+and+Michiel+Van+Lambalgen%2E+2012%2E+Human+Reasoning+and+Cognitive+Science%2E+MIT+Press%2E)
23. Johan van Benthem. 1986. Essays in Logical Semantics. Reidel, Dordrecht. [find](https://scholar.google.com/scholar?q=Johan+van+Benthem%2E+1986%2E+Essays+in+Logical+Semantics%2E+Reidel%2C+Dordrecht%2E)
24. Minna Cheves Wilkins. 1928. The effect of changed material on ability to do formal syllogistic reasoning. Archives of Psychology, 16:1–83. [find](https://scholar.google.com/scholar?q=Minna+Cheves+Wilkins%2E+1928%2E+The+effect+of+changed+material+on+ability+to+do+formal+syllogistic+reasoning%2E+Archives+of+Psychology%2C+16%3A1%E2%80%9383%2E)
25. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa…
26. Robert S Woodworth and Saul B. Sells. 1935. An atmosphere effect in formal syllogistic reasoning. Journal of Experimental Psychology, 18(4):451–460. [find](https://scholar.google.com/scholar?q=Robert+S+Woodworth+and+Saul+B%2E+Sells%2E+1935%2E+An+atmosphere+effect+in+formal+syllogistic+reasoning%2E+Journal+of+Experimental+Psychology%2C+18%284%29%3A451%E2%80%93460%2E)
27. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019. Can neural networks understand monotonicity reasoning? In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 31–40. [find](https://scholar.google.com/scholar?q=Hitomi+Yanaka%2C+Koji+Mineshima%2C+Daisuke+Bekki%2C+Kentaro+Inui%2C+Satoshi+Sekine%2C+Lasha+Abzianidze%2C+and+Johan+Bos%2E+2019%2E+Can+neural+networks+understand+monotonicity+reasoning%3F+In+Proceedings+of+the+2019+ACL)

*Updated 31 October 2026*