--- title: "SciBench: Evaluating College-Level Scientific — Wang 2023" description: "Benchmarks for scientific reasoning in LLMs have mostly used school-level questions that need only simple arithmetic, and they rarely include figures." authors: ["Xiaoxuan Wang", "Ziniu Hu", "Pan Lu", "Yanqiao Zhu", "Jieyu Zhang", "Satyen Subramaniam", "Arjun R. Loomba", "Shichang Zhang", "Yizhou Sun", "Wei Wang"] year: 2023 venue: "arXiv:2307.10635" source: "https://arxiv.org/abs/2307.10635" doi: "10.48550/arxiv.2307.10635" arxiv: "2307.10635" licence: "arXiv — non-exclusive licence to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/wang-2023-scibench-evaluating-college-level-scientific/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2023-scibench-evaluating-college-level-scientific/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models # SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models **Authors:** Xiaoxuan Wang, [Ziniu Hu](https://md.research.elamur.ai/free/en/authors/ziniu-hu/), [Pan Lu](https://md.research.elamur.ai/free/en/authors/pan-lu/), Yanqiao Zhu, [Jieyu Zhang](https://md.research.elamur.ai/free/en/authors/jieyu-zhang/), Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, [Wei Wang](https://md.research.elamur.ai/free/en/authors/wei-wang/) · **Year:** 2023 · **Venue:** arXiv:2307.10635 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2307.10635) · [PDF](https://arxiv.org/pdf/2307.10635) · [DOI 10.48550/arxiv.2307.10635](https://doi.org/10.48550/arxiv.2307.10635) · [arXiv 2307.10635](https://arxiv.org/abs/2307.10635) · [OpenAlex](https://openalex.org/W4385002382) **Citations:** 19 citations (openalex-snapshot · 2026-10-14) · [129](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Benchmarks for scientific reasoning in LLMs have mostly used school-level questions that need only simple arithmetic, and they rarely include figures. That makes models look stronger at science than they are. SciBench fills this gap with a curated set of college-level problems in mathematics, chemistry and physics, plus exam questions from computer science and mathematics. It includes both text-only and multimodal items. The authors test open-source and proprietary LLMs under several prompting setups: zero-shot, few-shot, Chain-of-Thought (CoT) and use of external tools. The best configuration reaches only 43.22% overall. To find out why models fail, the authors define ten problem-solving skills and attribute each error to a missing skill. GPT-3.5 does this labeling through a self-critique protocol, and human annotators then check its decisions. No prompting strategy is clearly best. Each one strengthens some skills and weakens others. The benchmark and its error analysis give a harder and more detailed way to measure scientific reasoning. Dataset and code are public. ## Key points - SciBench covers college-level mathematics, chemistry and physics problems, plus CS and math exam questions, with text-only and multimodal settings. It was published at ICML 2024 and is hosted at scibench-ucla.github.io. - The best model/prompting combination scores only 43.22% overall. - In error attribution, GPT-3.5 labels which of ten skills each wrong answer lacked. Human review then removes about 20% of these labels as misclassified. - Zero-shot CoT cuts calculation errors from 29.0% to 13.6%. It also increases causal-reasoning errors (18.3% → 32.2%) and logical-decomposition errors (18.3% → 25.4%). - External tools reduce calculation errors from 14.5% (few-shot CoT) to 6.2%. They hurt code-conversion skills, meaning the model is worse at writing correct programs for the solution. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/) ## Cited by Cited by 129 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 119 not yet. - [Nemotron-4 340B Technical Report](https://md.research.elamur.ai/free/en/papers/adler-2024-nemotron-4-340b-technical-report/) - [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://md.research.elamur.ai/free/en/papers/he-2024-olympiadbench-challenging-benchmark-promoting-agi/) - [Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models](https://md.research.elamur.ai/free/en/papers/lu-2023-chameleon-plug-play-compositional-reasoning/) - [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/) - [InFoBench: Evaluating Instruction Following Ability in Large Language Models](https://md.research.elamur.ai/free/en/papers/qin-2024-infobench-evaluating-instruction-following-ability/) - [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/) - [MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/) - [MAmmoTH2: Scaling Instructions from the Web](https://md.research.elamur.ai/free/en/papers/yue-2024-mammoth2-scaling-instructions-web/) - [ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search](https://md.research.elamur.ai/free/en/papers/zhang-2024-rest-mcts-llm-self-training/) - [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{wang2023scibench, title = {{SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models}}, author = {Wang, Xiaoxuan and Hu, Ziniu and Lu, Pan and Zhu, Yanqiao and Zhang, Jieyu and Subramaniam, Satyen and Loomba, Arjun R. and Zhang, Shichang and Sun, Yizhou and Wang, Wei}, year = {2023}, doi = {10.48550/arxiv.2307.10635}, eprint = {2307.10635}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2307.10635} } ``` - [← previous Scaling up CometKiwi: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task](https://md.research.elamur.ai/free/en/papers/rei-2023-scaling-up-cometkiwi-unbabel-ist/) - [next → Scoring Coreference Chains with Split-Antecedent Anaphors](https://md.research.elamur.ai/free/en/papers/paun-2023-scoring-coreference-chains-split-antecedent/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) ## References 62 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Anthropic. Claude2. [https://www.anthropic.com/index/claude-2](https://www.anthropic.com/index/claude-2), 2023. 5 [↗](https://www.anthropic.com/index/claude-2) 2. Arora, D., Singh, H. G., et al. Have llms advanced enough? a challenging problem solving benchmark for large language models. arXiv preprint arXiv:2305.15074, 2023. 1, 3 [find](https://scholar.google.com/scholar?q=Arora%2C+D%2E%2C+Singh%2C+H%2E+G%2E%2C+et+al%2E+Have+llms+advanced+enough%3F+a+challenging+problem+solving+benchmark+for+large+language+models%2E+arXiv+preprint+arXiv%3A2305%2E15074%2C+2023%2E+1%2C+3) 3. Atkins, P., Atkins, P. W., and de Paula, J. Atkins’ physical chemistry. Oxford university press, 2014a. 4, 12 [find](https://scholar.google.com/scholar?q=Atkins%2C+P%2E%2C+Atkins%2C+P%2E+W%2E%2C+and+de+Paula%2C+J%2E+Atkins%E2%80%99+physical+chemistry%2E+Oxford+university+press%2C+2014a%2E+4%2C+12) 4. Atkins, P., De Paula, J., and Friedman, R. Physical chemistry: quanta, matter, and change. Oxford University Press, USA, 2014b. 2, 4, 12 [find](https://scholar.google.com/scholar?q=Atkins%2C+P%2E%2C+De+Paula%2C+J%2E%2C+and+Friedman%2C+R%2E+Physical+chemistry%3A+quanta%2C+matter%2C+and+change%2E+Oxford+University+Press%2C+USA%2C+2014b%2E+2%2C+4%2C+12) 5. Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. 7 [find](https://scholar.google.com/scholar?q=Bai%2C+J%2E%2C+Bai%2C+S%2E%2C+Yang%2C+S%2E%2C+Wang%2C+S%2E%2C+Tan%2C+S%2E%2C+Wang%2C+P%2E%2C+Lin%2C+J%2E%2C+Zhou%2C+C%2E%2C+and+Zhou%2C+J%2E+Qwen%2Dvl%3A+A+versatile+vision%2Dlanguage+model+for+understanding%2C+localization%2C+text+reading%2C+and+beyond%2E+arXiv+pre) [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/) 6. Boyce, W. E., DiPrima, R. C., and Meade, D. B. Elementary differential equations and boundary value problems. John Wiley & Sons, 2021. 4, 12 [find](https://scholar.google.com/scholar?q=Boyce%2C+W%2E+E%2E%2C+DiPrima%2C+R%2E+C%2E%2C+and+Meade%2C+D%2E+B%2E+Elementary+differential+equations+and+boundary+value+problems%2E+John+Wiley+%26+Sons%2C+2021%2E+4%2C+12) 7. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 1 [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E%2C+Mann%2C+B%2E%2C+Ryder%2C+N%2E%2C+Subbiah%2C+M%2E%2C+Kaplan%2C+J%2E+D%2E%2C+Dhariwal%2C+P%2E%2C+Neelakantan%2C+A%2E%2C+Shyam%2C+P%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+Advances+in+neural+informati) 8. Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. 4 [find](https://scholar.google.com/scholar?q=Bubeck%2C+S%2E%2C+Chandrasekaran%2C+V%2E%2C+Eldan%2C+R%2E%2C+Gehrke%2C+J%2E%2C+Horvitz%2C+E%2E%2C+Kamar%2C+E%2E%2C+Lee%2C+P%2E%2C+Lee%2C+Y%2E+T%2E%2C+Li%2C+Y%2E%2C+Lundberg%2C+S%2E%2C+et+al%2E+Sparks+of+artificial+general+intelligence%3A+Early+experiments+with+gpt%2D4) [our card](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/) 9. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. 1 [find](https://scholar.google.com/scholar?q=Chen%2C+M%2E%2C+Tworek%2C+J%2E%2C+Jun%2C+H%2E%2C+Yuan%2C+Q%2E%2C+Pinto%2C+H%2E+P%2E+d%2E+O%2E%2C+Kaplan%2C+J%2E%2C+Edwards%2C+H%2E%2C+Burda%2C+Y%2E%2C+Joseph%2C+N%2E%2C+Brockman%2C+G%2E%2C+et+al%2E+Evaluating+large+language+models+trained+on+code%2E+arXiv+preprint+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/) 10. Chen, W., Ma, X., Wang, X., and Cohen, W. W. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research (TMLR), 2023a. 1, 5, 7 [find](https://scholar.google.com/scholar?q=Chen%2C+W%2E%2C+Ma%2C+X%2E%2C+Wang%2C+X%2E%2C+and+Cohen%2C+W%2E+W%2E+Program+of+thoughts+prompting%3A+Disentangling+computation+from+reasoning+for+numerical+reasoning+tasks%2E+Transactions+on+Machine+Learning+Research+%28TMLR%29%2C+20) 11. Chen, W., Yin, M., Ku, M., Lu, P., Wan, E., Ma, X., Xu, J., Xia, T., and Wang, X. Theoremqa: A theorem-driven question answering dataset. arXiv preprint arXiv:2305.12524, 2023b. 3, 5 [find](https://scholar.google.com/scholar?q=Chen%2C+W%2E%2C+Yin%2C+M%2E%2C+Ku%2C+M%2E%2C+Lu%2C+P%2E%2C+Wan%2C+E%2E%2C+Ma%2C+X%2E%2C+Xu%2C+J%2E%2C+Xia%2C+T%2E%2C+and+Wang%2C+X%2E+Theoremqa%3A+A+theorem%2Ddriven+question+answering+dataset%2E+arXiv+preprint+arXiv%3A2305%2E12524%2C+2023b%2E+3%2C+5) [our card](https://md.research.elamur.ai/free/en/papers/chen-2023-theoremqa-theorem-driven-question-answering/) 12. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. 1, 3, 4 [find](https://scholar.google.com/scholar?q=Cobbe%2C+K%2E%2C+Kosaraju%2C+V%2E%2C+Bavarian%2C+M%2E%2C+Chen%2C+M%2E%2C+Jun%2C+H%2E%2C+Kaiser%2C+L%2E%2C+Plappert%2C+M%2E%2C+Tworek%2C+J%2E%2C+Hilton%2C+J%2E%2C+Nakano%2C+R%2E%2C+et+al%2E+Training+verifiers+to+solve+math+word+problems%2E+arXiv+preprint+arXiv%3A2110) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/) 13. Dong, X., Zhang, P., Zang, Y., Cao, Y., Wang, B., Ouyang, L., Wei, X., Zhang, S., Duan, H., Cao, M., Zhang, W., Li, Y., Yan, H., Gao, Y., Zhang, X., Li, W., Li, J., Chen, K., He, C., Zhang, X., Qiao, Y., Lin, D., and Wang, J. Internlm-xcomposer2: Mastering free-form text-image composition and compre… [find](https://scholar.google.com/scholar?q=Dong%2C+X%2E%2C+Zhang%2C+P%2E%2C+Zang%2C+Y%2E%2C+Cao%2C+Y%2E%2C+Wang%2C+B%2E%2C+Ouyang%2C+L%2E%2C+Wei%2C+X%2E%2C+Zhang%2C+S%2E%2C+Duan%2C+H%2E%2C+Cao%2C+M%2E%2C+Zhang%2C+W%2E%2C+Li%2C+Y%2E%2C+Yan%2C+H%2E%2C+Gao%2C+Y%2E%2C+Zhang%2C+X%2E%2C+Li%2C+W%2E%2C+Li%2C+J%2E%2C+Chen%2C+K%2E%2C+He%2C+C%2E%2C+Zhang%2C+X%2E%2C+Qiao%2C+) [our card](https://md.research.elamur.ai/free/en/papers/dong-2024-internlm-xcomposer2-mastering-free/) 14. Engel, T. and Reid, P. J. Thermodynamics, statistical thermodynamics, and kinetics. Prentice Hall Upper saddle River, 2010. 4, 12 [find](https://scholar.google.com/scholar?q=Engel%2C+T%2E+and+Reid%2C+P%2E+J%2E+Thermodynamics%2C+statistical+thermodynamics%2C+and+kinetics%2E+Prentice+Hall+Upper+saddle+River%2C+2010%2E+4%2C+12) 15. Frieder, S., Pinchetti, L., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P. C., Chevalier, A., and Berner, J. Mathematical capabilities of chatgpt. arXiv preprint arXiv:2301.13867, 2023. 4 [find](https://scholar.google.com/scholar?q=Frieder%2C+S%2E%2C+Pinchetti%2C+L%2E%2C+Griffiths%2C+R%2E%2DR%2E%2C+Salvatori%2C+T%2E%2C+Lukasiewicz%2C+T%2E%2C+Petersen%2C+P%2E+C%2E%2C+Chevalier%2C+A%2E%2C+and+Berner%2C+J%2E+Mathematical+capabilities+of+chatgpt%2E+arXiv+preprint+arXiv%3A2301%2E13867%2C+2023) [our card](https://md.research.elamur.ai/free/en/papers/frieder-2023-mathematical-capabilities-chatgpt/) 16. Fu, Y., Ou, L., Chen, M., Wan, Y., Peng, H., and Khot, T. Chain-of-thought hub: A continuous effort to measure large language models’ reasoning performance. arXiv preprint arXiv:2305.17306, 2023. 3 [find](https://scholar.google.com/scholar?q=Fu%2C+Y%2E%2C+Ou%2C+L%2E%2C+Chen%2C+M%2E%2C+Wan%2C+Y%2E%2C+Peng%2C+H%2E%2C+and+Khot%2C+T%2E+Chain%2Dof%2Dthought+hub%3A+A+continuous+effort+to+measure+large+language+models%E2%80%99+reasoning+performance%2E+arXiv+preprint+arXiv%3A2305%2E17306%2C+2023%2E+3) 17. Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. PAL: Program-aided language models. arXiv preprint arXiv:2211.10435, 2022. 1, 5 [find](https://scholar.google.com/scholar?q=Gao%2C+L%2E%2C+Madaan%2C+A%2E%2C+Zhou%2C+S%2E%2C+Alon%2C+U%2E%2C+Liu%2C+P%2E%2C+Yang%2C+Y%2E%2C+Callan%2C+J%2E%2C+and+Neubig%2C+G%2E+PAL%3A+Program%2Daided+language+models%2E+arXiv+preprint+arXiv%3A2211%2E10435%2C+2022%2E+1%2C+5) 18. Gao, P., Han, J., Zhang, R., Lin, Z., Geng, S., Zhou, A., Zhang, W., Lu, P., He, C., Yue, X., Li, H., and Qiao, Y. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010, 2023. 1 [find](https://scholar.google.com/scholar?q=Gao%2C+P%2E%2C+Han%2C+J%2E%2C+Zhang%2C+R%2E%2C+Lin%2C+Z%2E%2C+Geng%2C+S%2E%2C+Zhou%2C+A%2E%2C+Zhang%2C+W%2E%2C+Lu%2C+P%2E%2C+He%2C+C%2E%2C+Yue%2C+X%2E%2C+Li%2C+H%2E%2C+and+Qiao%2C+Y%2E+Llama%2Dadapter+v2%3A+Parameter%2Defficient+visual+instruction+model%2E+arXiv+preprint+arXiv%3A) 19. Ghazal, A., Rabl, T., Hu, M., Raab, F., Poess, M., Crolotte, A., and Jacobsen, H.-A. Bigbench: Towards an industry standard benchmark for big data analytics. In Proceedings of the 2013 ACM SIGMOD international conference on Management of data, pp. 1197–1208, 2013. 3 [find](https://scholar.google.com/scholar?q=Ghazal%2C+A%2E%2C+Rabl%2C+T%2E%2C+Hu%2C+M%2E%2C+Raab%2C+F%2E%2C+Poess%2C+M%2E%2C+Crolotte%2C+A%2E%2C+and+Jacobsen%2C+H%2E%2DA%2E+Bigbench%3A+Towards+an+industry+standard+benchmark+for+big+data+analytics%2E+In+Proceedings+of+the+2013+ACM+SIGMOD+inte) 20. Google. Bard. [https://bard.google.com](https://bard.google.com), 2023. 7 [↗](https://bard.google.com) 21. Guo, T., Guo, K., Liang, Z., Guo, Z., Chawla, N. V., Wiest, O., Zhang, X., et al. What indeed can gpt models do in chemistry? a comprehensive benchmark on eight tasks. arXiv preprint arXiv:2305.18365, 2023. 3 — [arXiv:2305.18365](https://arxiv.org/abs/2305.18365) [find](https://scholar.google.com/scholar?q=Guo%2C+T%2E%2C+Guo%2C+K%2E%2C+Liang%2C+Z%2E%2C+Guo%2C+Z%2E%2C+Chawla%2C+N%2E+V%2E%2C+Wiest%2C+O%2E%2C+Zhang%2C+X%2E%2C+et+al%2E+What+indeed+can+gpt+models+do+in+chemistry%3F+a+comprehensive+benchmark+on+eight+tasks%2E+arXiv+preprint+arXiv%3A2305%2E18365%2C) 22. Halliday, D., Resnick, R., and Walker, J. Fundamentals of physics. John Wiley & Sons, 2013. 4, 12 [find](https://scholar.google.com/scholar?q=Halliday%2C+D%2E%2C+Resnick%2C+R%2E%2C+and+Walker%2C+J%2E+Fundamentals+of+physics%2E+John+Wiley+%26+Sons%2C+2013%2E+4%2C+12) 23. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. 1, 3 [find](https://scholar.google.com/scholar?q=Hendrycks%2C+D%2E%2C+Burns%2C+C%2E%2C+Basart%2C+S%2E%2C+Zou%2C+A%2E%2C+Mazeika%2C+M%2E%2C+Song%2C+D%2E%2C+and+Steinhardt%2C+J%2E+Measuring+massive+multitask+language+understanding%2E+arXiv+preprint+arXiv%3A2009%2E03300%2C+2020%2E+1%2C+3) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/) 24. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. 1, 3, 4 [find](https://scholar.google.com/scholar?q=Hendrycks%2C+D%2E%2C+Burns%2C+C%2E%2C+Kadavath%2C+S%2E%2C+Arora%2C+A%2E%2C+Basart%2C+S%2E%2C+Tang%2C+E%2E%2C+Song%2C+D%2E%2C+and+Steinhardt%2C+J%2E+Measuring+mathematical+problem+solving+with+the+math+dataset%2E+arXiv+preprint+arXiv%3A2103%2E03874%2C+202) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/) 25. Hogg, R. V., Tanis, E. A., and Zimmerman, D. L. Probability and statistical inference, volume 993. Macmillan New York, 1977. 4, 13 [find](https://scholar.google.com/scholar?q=Hogg%2C+R%2E+V%2E%2C+Tanis%2C+E%2E+A%2E%2C+and+Zimmerman%2C+D%2E+L%2E+Probability+and+statistical+inference%2C+volume+993%2E+Macmillan+New+York%2C+1977%2E+4%2C+13) 26. Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. Large language models can self-improve. arXiv preprint arXiv:2210.11610, 2022. 1 [find](https://scholar.google.com/scholar?q=Huang%2C+J%2E%2C+Gu%2C+S%2E+S%2E%2C+Hou%2C+L%2E%2C+Wu%2C+Y%2E%2C+Wang%2C+X%2E%2C+Yu%2C+H%2E%2C+and+Han%2C+J%2E+Large+language+models+can+self%2Dimprove%2E+arXiv+preprint+arXiv%3A2210%2E11610%2C+2022%2E+1) [our card](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/) 27. JaidedAI. Easyocr: Ready-to-use ocr. [https://github.com/J](https://github.com/J) aidedAI/EasyOCR, 2022. 7 [↗](https://github.com/J) 28. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. 5 [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E%2C+Sablayrolles%2C+A%2E%2C+Mensch%2C+A%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2E+S%2E%2C+Casas%2C+D%2E+d%2E+l%2E%2C+Bressand%2C+F%2E%2C+Lengyel%2C+G%2E%2C+Lample%2C+G%2E%2C+Saulnier%2C+L%2E%2C+et+al%2E+Mistral+7b%2E+arXiv+preprint+arXiv%3A2310%2E06825%2C+2023%2E) 29. Kabir, S., Udo-Imeh, D. N., Kou, B., and Zhang, T. Who answers it better? an in-depth analysis of chatgpt and stack overflow answers to software engineering questions. arXiv preprint arXiv:2308.02312, 2023. 4 — [arXiv:2308.02312](https://arxiv.org/abs/2308.02312) [find](https://scholar.google.com/scholar?q=Kabir%2C+S%2E%2C+Udo%2DImeh%2C+D%2E+N%2E%2C+Kou%2C+B%2E%2C+and+Zhang%2C+T%2E+Who+answers+it+better%3F+an+in%2Ddepth+analysis+of+chatgpt+and+stack+overflow+answers+to+software+engineering+questions%2E+arXiv+preprint+arXiv%3A2308%2E02312%2C) 30. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022. 1 [find](https://scholar.google.com/scholar?q=Kojima%2C+T%2E%2C+Gu%2C+S%2E+S%2E%2C+Reid%2C+M%2E%2C+Matsuo%2C+Y%2E%2C+and+Iwasawa%2C+Y%2E+Large+language+models+are+zero%2Dshot+reasoners%2E+arXiv+preprint+arXiv%3A2205%2E11916%2C+2022%2E+1) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/) 31. Levine, I. N., Busch, D. H., and Shull, H. Quantum chemistry, volume 6. Pearson Prentice Hall Upper Saddle River, NJ, 2009. 4, 12 [find](https://scholar.google.com/scholar?q=Levine%2C+I%2E+N%2E%2C+Busch%2C+D%2E+H%2E%2C+and+Shull%2C+H%2E+Quantum+chemistry%2C+volume+6%2E+Pearson+Prentice+Hall+Upper+Saddle+River%2C+NJ%2C+2009%2E+4%2C+12) 32. Lin, Z., Liu, C., Zhang, R., Gao, P., Qiu, L., Xiao, H., Qiu, H., Lin, C., Shao, W., Chen, K., et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575, 2023. 7 — [arXiv:2311.07575](https://arxiv.org/abs/2311.07575) [find](https://scholar.google.com/scholar?q=Lin%2C+Z%2E%2C+Liu%2C+C%2E%2C+Zhang%2C+R%2E%2C+Gao%2C+P%2E%2C+Qiu%2C+L%2E%2C+Xiao%2C+H%2E%2C+Qiu%2C+H%2E%2C+Lin%2C+C%2E%2C+Shao%2C+W%2E%2C+Chen%2C+K%2E%2C+et+al%2E+Sphinx%3A+The+joint+mixing+of+weights%2C+tasks%2C+and+visual+embeddings+for+multi%2Dmodal+large+language+m) 33. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. In NeurIPS, 2023a. 7 [find](https://scholar.google.com/scholar?q=Liu%2C+H%2E%2C+Li%2C+C%2E%2C+Wu%2C+Q%2E%2C+and+Lee%2C+Y%2E+J%2E+Visual+instruction+tuning%2E+In+NeurIPS%2C+2023a%2E+7) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/) 34. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023b. 1 [find](https://scholar.google.com/scholar?q=Liu%2C+H%2E%2C+Li%2C+C%2E%2C+Wu%2C+Q%2E%2C+and+Lee%2C+Y%2E+J%2E+Visual+instruction+tuning%2E+arXiv+preprint+arXiv%3A2304%2E08485%2C+2023b%2E+1) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/) 35. Liu, H., Ning, R., Teng, Z., Liu, J., Zhou, Q., and Zhang, Y. Evaluating the logical reasoning ability of chatgpt and gpt-4. arXiv preprint arXiv:2304.03439, 2023c. 4 [find](https://scholar.google.com/scholar?q=Liu%2C+H%2E%2C+Ning%2C+R%2E%2C+Teng%2C+Z%2E%2C+Liu%2C+J%2E%2C+Zhou%2C+Q%2E%2C+and+Zhang%2C+Y%2E+Evaluating+the+logical+reasoning+ability+of+chatgpt+and+gpt%2D4%2E+arXiv+preprint+arXiv%3A2304%2E03439%2C+2023c%2E+4) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-evaluating-logical-reasoning-ability-chatgpt/) 36. Lu, P., Gong, R., Jiang, S., Qiu, L., Huang, S., Liang, X., and Zhu, S.-C. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In The Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joi… [find](https://scholar.google.com/scholar?q=Lu%2C+P%2E%2C+Gong%2C+R%2E%2C+Jiang%2C+S%2E%2C+Qiu%2C+L%2E%2C+Huang%2C+S%2E%2C+Liang%2C+X%2E%2C+and+Zhu%2C+S%2E%2DC%2E+Inter%2Dgps%3A+Interpretable+geometry+problem+solving+with+formal+language+and+symbolic+reasoning%2E+In+The+Joint+Conference+of+the) [our card](https://md.research.elamur.ai/free/en/papers/lu-2021-inter-gps-interpretable-geometry-problem/) 37. Lu, P., Qiu, L., Chen, J., Xia, T., Zhao, Y., Zhang, W., Yu, Z., Liang, X., and Zhu, S.-C. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2110.13214, 2021b. 3 [find](https://scholar.google.com/scholar?q=Lu%2C+P%2E%2C+Qiu%2C+L%2E%2C+Chen%2C+J%2E%2C+Xia%2C+T%2E%2C+Zhao%2C+Y%2E%2C+Zhang%2C+W%2E%2C+Yu%2C+Z%2E%2C+Liang%2C+X%2E%2C+and+Zhu%2C+S%2E%2DC%2E+Iconqa%3A+A+new+benchmark+for+abstract+diagram+understanding+and+visual+language+reasoning%2E+arXiv+preprint+arXi) 38. Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022. 1, 3, 5 [find](https://scholar.google.com/scholar?q=Lu%2C+P%2E%2C+Mishra%2C+S%2E%2C+Xia%2C+T%2E%2C+Qiu%2C+L%2E%2C+Chang%2C+K%2E%2DW%2E%2C+Zhu%2C+S%2E%2DC%2E%2C+Tafjord%2C+O%2E%2C+Clark%2C+P%2E%2C+and+Kalyan%2C+A%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+science+question+answering%2E+Advance) 39. Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023a. 3 [find](https://scholar.google.com/scholar?q=Lu%2C+P%2E%2C+Bansal%2C+H%2E%2C+Xia%2C+T%2E%2C+Liu%2C+J%2E%2C+Li%2C+C%2E%2C+Hajishirzi%2C+H%2E%2C+Cheng%2C+H%2E%2C+Chang%2C+K%2E%2DW%2E%2C+Galley%2C+M%2E%2C+and+Gao%2C+J%2E+Mathvista%3A+Evaluating+mathematical+reasoning+of+foundation+models+in+visual+contexts%2E+arX) [our card](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/) 40. Lu, P., Peng, B., Cheng, H., Galley, M., Chang, K.-W., Wu, Y. N., Zhu, S.-C., and Gao, J. Chameleon: Plug-and-play compositional reasoning with large language models. arXiv preprint arXiv:2304.09842, 2023b. 1 [find](https://scholar.google.com/scholar?q=Lu%2C+P%2E%2C+Peng%2C+B%2E%2C+Cheng%2C+H%2E%2C+Galley%2C+M%2E%2C+Chang%2C+K%2E%2DW%2E%2C+Wu%2C+Y%2E+N%2E%2C+Zhu%2C+S%2E%2DC%2E%2C+and+Gao%2C+J%2E+Chameleon%3A+Plug%2Dand%2Dplay+compositional+reasoning+with+large+language+models%2E+arXiv+preprint+arXiv%3A2304%2E09842%2C+) [our card](https://md.research.elamur.ai/free/en/papers/lu-2023-chameleon-plug-play-compositional-reasoning/) 41. Lu, P., Qiu, L., Chang, K.-W., Wu, Y. N., Zhu, S.-C., Rajpurohit, T., Clark, P., and Kalyan, A. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In International Conference on Learning Representations (ICLR), 2023c. 3 [find](https://scholar.google.com/scholar?q=Lu%2C+P%2E%2C+Qiu%2C+L%2E%2C+Chang%2C+K%2E%2DW%2E%2C+Wu%2C+Y%2E+N%2E%2C+Zhu%2C+S%2E%2DC%2E%2C+Rajpurohit%2C+T%2E%2C+Clark%2C+P%2E%2C+and+Kalyan%2C+A%2E+Dynamic+prompt+learning+via+policy+gradient+for+semi%2Dstructured+mathematical+reasoning%2E+In+International) [our card](https://md.research.elamur.ai/free/en/papers/lu-2022-dynamic-prompt-learning-policy-gradient/) 42. Lu, P., Qiu, L., Yu, W., Welleck, S., and Chang, K.-W. A survey of deep learning for mathematical reasoning. In The 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023d. 3 [find](https://scholar.google.com/scholar?q=Lu%2C+P%2E%2C+Qiu%2C+L%2E%2C+Yu%2C+W%2E%2C+Welleck%2C+S%2E%2C+and+Chang%2C+K%2E%2DW%2E+A+survey+of+deep+learning+for+mathematical+reasoning%2E+In+The+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%29%2C+2023d%2E+3) 43. McQuarrie, D. A. Quantum chemistry. University Science Books, 2008. 4, 12 [find](https://scholar.google.com/scholar?q=McQuarrie%2C+D%2E+A%2E+Quantum+chemistry%2E+University+Science+Books%2C+2008%2E+4%2C+12) 44. Mishra, S., Finlayson, M., Lu, P., Tang, L., Welleck, S., Baral, C., Rajpurohit, T., Tafjord, O., Sabharwal, A., Clark, P., et al. Lila: A unified benchmark for mathematical reasoning. In The 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022. 3 [find](https://scholar.google.com/scholar?q=Mishra%2C+S%2E%2C+Finlayson%2C+M%2E%2C+Lu%2C+P%2E%2C+Tang%2C+L%2E%2C+Welleck%2C+S%2E%2C+Baral%2C+C%2E%2C+Rajpurohit%2C+T%2E%2C+Tafjord%2C+O%2E%2C+Sabharwal%2C+A%2E%2C+Clark%2C+P%2E%2C+et+al%2E+Lila%3A+A+unified+benchmark+for+mathematical+reasoning%2E+In+The+2022+Con) [our card](https://md.research.elamur.ai/free/en/papers/mishra-2022-lila-unified-benchmark-mathematical-reasoning/) 45. OpenAI. Chatgpt: Optimizing language models for dialogue. [https://openai.com/blog/chatgpt/](https://openai.com/blog/chatgpt/)., 2022. 1, 5 [↗](https://openai.com/blog/chatgpt/) 46. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 1, 5, 7 [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2E+arXiv+preprint+arXiv%3A2303%2E08774%2C+2023%2E+1%2C+5%2C+7) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/) 47. Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023. 1 [find](https://scholar.google.com/scholar?q=Schick%2C+T%2E%2C+Dwivedi%2DYu%2C+J%2E%2C+Dess%C3%AC%2C+R%2E%2C+Raileanu%2C+R%2E%2C+Lomeli%2C+M%2E%2C+Zettlemoyer%2C+L%2E%2C+Cancedda%2C+N%2E%2C+and+Scialom%2C+T%2E+Toolformer%3A+Language+models+can+teach+themselves+to+use+tools%2E+arXiv+preprint+arXiv%3A2302) [our card](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/) 48. Stewart, J., Watson, S., and Clegg, D. Calculus: Early transcendentals, 8th. Edition, Brooks/Cole, Cengae learning, 2012. 4, 13 [find](https://scholar.google.com/scholar?q=Stewart%2C+J%2E%2C+Watson%2C+S%2E%2C+and+Clegg%2C+D%2E+Calculus%3A+Early+transcendentals%2C+8th%2E+Edition%2C+Brooks%2FCole%2C+Cengae+learning%2C+2012%2E+4%2C+13) 49. Sun, L., Han, Y., Zhao, Z., Ma, D., Shen, Z., Chen, B., Chen, L., and Yu, K. Scieval: A multi-level large language model evaluation benchmark for scientific research. arXiv preprint arXiv:2308.13149, 2023. 3 [find](https://scholar.google.com/scholar?q=Sun%2C+L%2E%2C+Han%2C+Y%2E%2C+Zhao%2C+Z%2E%2C+Ma%2C+D%2E%2C+Shen%2C+Z%2E%2C+Chen%2C+B%2E%2C+Chen%2C+L%2E%2C+and+Yu%2C+K%2E+Scieval%3A+A+multi%2Dlevel+large+language+model+evaluation+benchmark+for+scientific+research%2E+arXiv+preprint+arXiv%3A2308%2E13149%2C+) 50. Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022. 3 [find](https://scholar.google.com/scholar?q=Suzgun%2C+M%2E%2C+Scales%2C+N%2E%2C+Sch%C3%A4rli%2C+N%2E%2C+Gehrmann%2C+S%2E%2C+Tay%2C+Y%2E%2C+Chung%2C+H%2E+W%2E%2C+Chowdhery%2C+A%2E%2C+Le%2C+Q%2E+V%2E%2C+Chi%2C+E%2E+H%2E%2C+Zhou%2C+D%2E%2C+et+al%2E+Challenging+big%2Dbench+tasks+and+whether+chain%2Dof%2Dthought+can+solve+them) [our card](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/) 51. Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085, 2022. 3 [find](https://scholar.google.com/scholar?q=Taylor%2C+R%2E%2C+Kardas%2C+M%2E%2C+Cucurull%2C+G%2E%2C+Scialom%2C+T%2E%2C+Hartshorn%2C+A%2E%2C+Saravia%2C+E%2E%2C+Poulton%2C+A%2E%2C+Kerkez%2C+V%2E%2C+and+Stojnic%2C+R%2E+Galactica%3A+A+large+language+model+for+science%2E+arXiv+preprint+arXiv%3A2211%2E09085%2C+) [our card](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/) 52. Thornton, S. T. and Marion, J. B. Classical dynamics of particles and systems. Cengage Learning, 2021. 4, 12 [find](https://scholar.google.com/scholar?q=Thornton%2C+S%2E+T%2E+and+Marion%2C+J%2E+B%2E+Classical+dynamics+of+particles+and+systems%2E+Cengage+Learning%2C+2021%2E+4%2C+12) 53. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a. 1 [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%2C+Lavril%2C+T%2E%2C+Izacard%2C+G%2E%2C+Martinet%2C+X%2E%2C+Lachaux%2C+M%2E%2DA%2E%2C+Lacroix%2C+T%2E%2C+Rozi%C3%A8re%2C+B%2E%2C+Goyal%2C+N%2E%2C+Hambro%2C+E%2E%2C+Azhar%2C+F%2E%2C+et+al%2E+LLaMA%3A+Open+and+efficient+foundation+language+models%2E+arXiv+prepr) 54. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b. 5 [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%2C+Martin%2C+L%2E%2C+Stone%2C+K%2E%2C+Albert%2C+P%2E%2C+Almahairi%2C+A%2E%2C+Babaei%2C+Y%2E%2C+Bashlykov%2C+N%2E%2C+Batra%2C+S%2E%2C+Bhargava%2C+P%2E%2C+Bhosale%2C+S%2E%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtuned+chat+models%2E+arXiv+prepri) 55. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. 1 [find](https://scholar.google.com/scholar?q=Wang%2C+X%2E%2C+Wei%2C+J%2E%2C+Schuurmans%2C+D%2E%2C+Le%2C+Q%2E%2C+Chi%2C+E%2E%2C+and+Zhou%2C+D%2E+Self%2Dconsistency+improves+chain+of+thought+reasoning+in+language+models%2E+arXiv+preprint+arXiv%3A2203%2E11171%2C+2022%2E+1) [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/) 56. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. 1, 7, 8 [find](https://scholar.google.com/scholar?q=Wei%2C+J%2E%2C+Wang%2C+X%2E%2C+Schuurmans%2C+D%2E%2C+Bosma%2C+M%2E%2C+Chi%2C+E%2E%2C+Le%2C+Q%2E%2C+and+Zhou%2C+D%2E+Chain+of+thought+prompting+elicits+reasoning+in+large+language+models%2E+arXiv+preprint+arXiv%3A2201%2E11903%2C+2022%2E+1%2C+7%2C+8) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/) 57. Welleck, S., Liu, J., Bras, R. L., Hajishirzi, H., Choi, Y., and Cho, K. Naturalproofs: Mathematical theorem proving in natural language. arXiv preprint arXiv:2104.01112, 2021. 3 [find](https://scholar.google.com/scholar?q=Welleck%2C+S%2E%2C+Liu%2C+J%2E%2C+Bras%2C+R%2E+L%2E%2C+Hajishirzi%2C+H%2E%2C+Choi%2C+Y%2E%2C+and+Cho%2C+K%2E+Naturalproofs%3A+Mathematical+theorem+proving+in+natural+language%2E+arXiv+preprint+arXiv%3A2104%2E01112%2C+2021%2E+3) 58. Zhang, D., Hu, Z., Zhoubian, S., Du, Z., Yang, K., Wang, Z., Yue, Y., Dong, Y., and Tang, J. Sciglm: Training scientific language models with self-reflective instruction annotation and tuning. arXiv preprint arXiv:2401.07950, 2024. 4 [find](https://scholar.google.com/scholar?q=Zhang%2C+D%2E%2C+Hu%2C+Z%2E%2C+Zhoubian%2C+S%2E%2C+Du%2C+Z%2E%2C+Yang%2C+K%2E%2C+Wang%2C+Z%2E%2C+Yue%2C+Y%2E%2C+Dong%2C+Y%2E%2C+and+Tang%2C+J%2E+Sciglm%3A+Training+scientific+language+models+with+self%2Dreflective+instruction+annotation+and+tuning%2E+arXiv+p) 59. Zhang, R., Han, J., Zhou, A., Hu, X., Yan, S., Lu, P., Li, H., Gao, P., and Qiao, Y. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199, 2023a. 1 [find](https://scholar.google.com/scholar?q=Zhang%2C+R%2E%2C+Han%2C+J%2E%2C+Zhou%2C+A%2E%2C+Hu%2C+X%2E%2C+Yan%2C+S%2E%2C+Lu%2C+P%2E%2C+Li%2C+H%2E%2C+Gao%2C+P%2E%2C+and+Qiao%2C+Y%2E+Llama%2Dadapter%3A+Efficient+fine%2Dtuning+of+language+models+with+zero%2Dinit+attention%2E+arXiv+preprint+arXiv%3A2303%2E16199%2C+) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2023-llama-adapter-efficient-fine-tuning/) 60. Zhang, Z., Zhang, A., Li, M., Zhao, H., Karypis, G., and Smola, A. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023b. 1 [find](https://scholar.google.com/scholar?q=Zhang%2C+Z%2E%2C+Zhang%2C+A%2E%2C+Li%2C+M%2E%2C+Zhao%2C+H%2E%2C+Karypis%2C+G%2E%2C+and+Smola%2C+A%2E+Multimodal+chain%2Dof%2Dthought+reasoning+in+language+models%2E+arXiv+preprint+arXiv%3A2302%2E00923%2C+2023b%2E+1) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2023-multimodal-chain-thought-reasoning-language/) 61. Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., and Duan, N. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023. 1, 3, 7 [find](https://scholar.google.com/scholar?q=Zhong%2C+W%2E%2C+Cui%2C+R%2E%2C+Guo%2C+Y%2E%2C+Liang%2C+Y%2E%2C+Lu%2C+S%2E%2C+Wang%2C+Y%2E%2C+Saied%2C+A%2E%2C+Chen%2C+W%2E%2C+and+Duan%2C+N%2E+Agieval%3A+A+human%2Dcentric+benchmark+for+evaluating+foundation+models%2E+arXiv+preprint+arXiv%3A2304%2E06364%2C+2023%2E+) [our card](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/) 62. Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Bousquet, O., Le, Q., and Chi, E. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022. 1 [find](https://scholar.google.com/scholar?q=Zhou%2C+D%2E%2C+Sch%C3%A4rli%2C+N%2E%2C+Hou%2C+L%2E%2C+Wei%2C+J%2E%2C+Scales%2C+N%2E%2C+Wang%2C+X%2E%2C+Schuurmans%2C+D%2E%2C+Bousquet%2C+O%2E%2C+Le%2C+Q%2E%2C+and+Chi%2C+E%2E+Least%2Dto%2Dmost+prompting+enables+complex+reasoning+in+large+language+models%2E+arXiv+prepr) [our card](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/)