---
title: "PaSa: An LLM Agent for Comprehensive Academic Paper Search"
description: "Search tools like Google Scholar handle narrow, detailed research questions poorly, so… 85% F1. 32 references, cited by 20 papers here."
authors: ["Yichen He", "Guanhua Huang", "Peiyuan Feng", "Yuan Lin", "Yuchen Zhang", "Hang Li", "Weinan E"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.572/"
doi: "10.18653/v1/2025.acl-long.572"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/he-2025-pasa-llm-agent-comprehensive-academic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/he-2025-pasa-llm-agent-comprehensive-academic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/) / PaSa: An LLM Agent for Comprehensive Academic Paper Search

# PaSa: An LLM Agent for Comprehensive Academic Paper Search

**Authors:** Yichen He, [Guanhua Huang](https://md.research.elamur.ai/free/en/authors/guanhua-huang/), Peiyuan Feng, Yuan Lin, [Yuchen Zhang](https://md.research.elamur.ai/free/en/authors/yuchen-zhang/), [Hang Li](https://md.research.elamur.ai/free/en/authors/hang-li/) and [Weinan E](https://md.research.elamur.ai/free/en/authors/weinan-e/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.572/) · [PDF](https://aclanthology.org/2025.acl-long.572.pdf) · [DOI 10.18653/v1/2025.acl-long.572](https://doi.org/10.18653/v1/2025.acl-long.572) · [OpenAlex](https://openalex.org/W4412945469)

**Citations:** 9 citations (OpenAlex, 2026-10-14) · [20](https://md.research.elamur.ai/free/en/papers/he-2025-pasa-llm-agent-comprehensive-academic/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Search tools like Google Scholar handle narrow, detailed research questions poorly, so researchers spend long hours on manual literature surveys. PaSa addresses this with an agent made of two LLM components. The Crawler calls a search tool, follows the citations of papers it has already found, and decides when to stop, adding results to a growing paper queue. The Selector reads each queued paper and judges whether it fits the query. Both components run on a 7B model and are trained with reinforcement learning inside the AGILE framework. The authors use a new session-level PPO variant built to cope with sparse rewards and very long trajectories. Training data come from AutoScholarQuery, a synthetic set in which GPT-4o turned the Related Work sections of ICLR, ICML and NeurIPS 2023 and ACL and CVPR 2024 papers into queries. The answers to each query are the cited arXiv papers. A second benchmark, RealScholarQuery, holds 50 genuine researcher queries with annotated answers. Despite training only on synthetic data, PaSa-7B beats Google, Google Scholar, ChatGPT, GPT-o1 and a GPT-4o-prompted version of itself. This shows that compact RL-trained agents can make literature search much more complete.

## Key points
- **AutoScholarQuery** contains about 35k queries: roughly 33.5k for training, plus 1,000 for development and 1,000 for testing. A manual check of 100 pairs found 94.0% of queries well-formed, and 93.7% of their answer papers relevant.
- **RealScholarQuery** has 50 fine-grained queries from AI researchers using the PaSa demo. On it, PaSa-7B exceeds Google with GPT-4o by 37.78% in recall@20 and 39.90% in recall@50. Compared with PaSa-GPT-4o, it also achieves 30.36% higher recall and 4.25% higher precision.
- On the AutoScholarQuery test set, PaSa-7B gains 34.05% in Recall@20 and 39.36% in Recall@50 over Google with GPT-4o.
- The Selector reaches 85% F1 and 95% precision on 200 annotated query-paper pairs. This is 5 points above GPT-4o and 30 points above Qwen-2.5-7b.
- Ablations show the value of each component:
  - Dropping the citation-expansion action cuts recall by 22.98% on AutoScholarQuery and 32.21% on RealScholarQuery.
  - RL training adds 6.24% and 19.96% recall on the same two benchmarks.
- The model, data and code are released at github.com/bytedance/pasa. The work covers machine learning papers only and 7B models.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 244 works · 39,359 citations
- [ByteDance](https://md.research.elamur.ai/free/en/companies/bytedance/) — Company, Beijing, China · 41 works · 4,180 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

## Cite

Cite as: Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li and Weinan E. 2025. PaSa: An LLM Agent for Comprehensive Academic Paper Search. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{he2025pasa,
  title = {{PaSa: An LLM Agent for Comprehensive Academic Paper Search}},
  author = {Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li and Weinan E},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.572},
  url = {https://doi.org/10.18653/v1/2025.acl-long.572}
}
```

- [← previous INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent](https://md.research.elamur.ai/free/en/papers/li-2025-investorbench-benchmark-financial-decision-making/)
- [next → Persona-DB: Efficient Large Language Model Personalization for Response Prediction with Collaborative Data Refinement](https://md.research.elamur.ai/free/en/papers/sun-2025-persona-db-efficient-large-language/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/)

## References

32 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Shubham Agarwal, Issam H Laradji, Laurent Charlin, and Christopher Pal. 2024. Litllm: A toolkit for scientific literature review. arXiv preprint arXiv:2402.01788. [find](https://scholar.google.com/scholar?q=Shubham+Agarwal%2C+Issam+H+Laradji%2C+Laurent+Charlin%2C+and+Christopher+Pal%2E+2024%2E+Litllm%3A+A+toolkit+for+scientific+literature+review%2E+arXiv+preprint+arXiv%3A2402%2E01788%2E)
2. Marwah Alaofi, Luke Gallagher, Mark Sanderson, Falk Scholer, and Paul Thomas. 2023. Can generative llms create query variants for test collections? an exploratory study. In Proceedings of the 46th international ACM SIGIR conference on research and development in information retrieval, pages 1869– 18… [find](https://scholar.google.com/scholar?q=Marwah+Alaofi%2C+Luke+Gallagher%2C+Mark+Sanderson%2C+Falk+Scholer%2C+and+Paul+Thomas%2E+2023%2E+Can+generative+llms+create+query+variants+for+test+collections%3F+an+exploratory+study%2E+In+Proceedings+of+the+46th+int)
3. A Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku; 2024. URL [https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7](https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7) bbc618857627/Model_Card_Claude_3.pdf. [↗](https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7)
4. Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. 2024. [Researchagent: Iterative research idea generation over scientific literature with large language models](https://md.research.elamur.ai/free/en/papers/baek-2024-researchagent-iterative-research-idea/). arXiv preprint arXiv:2404.07738.
5. Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. 2023. Chemcrow: Augmenting large-language models with chemistry tools. arXiv preprint arXiv:2304.05376. [find](https://scholar.google.com/scholar?q=Andres+M+Bran%2C+Sam+Cox%2C+Oliver+Schilter%2C+Carlo+Baldassari%2C+Andrew+D+White%2C+and+Philippe+Schwaller%2E+2023%2E+Chemcrow%3A+Augmenting+large%2Dlanguage+models+with+chemistry+tools%2E+arXiv+preprint+arXiv%3A2304%2E0537)
6. Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F Karlsson, Jie Fu, and Yemin Shi. 2023. Autoagents: A framework for automatic agent generation. arXiv preprint arXiv:2309.17288. [find](https://scholar.google.com/scholar?q=Guangyao+Chen%2C+Siwei+Dong%2C+Yu+Shu%2C+Ge+Zhang%2C+Jaward+Sesay%2C+B%C3%B6rje+F+Karlsson%2C+Jie+Fu%2C+and+Yemin+Shi%2E+2023%2E+Autoagents%3A+A+framework+for+automatic+agent+generation%2E+arXiv+preprint+arXiv%3A2309%2E17288%2E)
7. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024. [Mind2web: Towards a generalist agent for the web](https://md.research.elamur.ai/free/en/papers/deng-2023-mind2web-towards-generalist-agent-web/). Advances in Neural Information Processing Systems, 36.
8. Peiyuan Feng, Yichen He, Guanhua Huang, Yuan Lin, Hanchong Zhang, Yuchen Zhang, and Hang Li. 2024. Agile: A novel reinforcement learning framework of llm agents. Advances in Neural Information Processing Systems, 37:5244–5284. [find](https://scholar.google.com/scholar?q=Peiyuan+Feng%2C+Yichen+He%2C+Guanhua+Huang%2C+Yuan+Lin%2C+Hanchong+Zhang%2C+Yuchen+Zhang%2C+and+Hang+Li%2E+2024%2E+Agile%3A+A+novel+reinforcement+learning+framework+of+llm+agents%2E+Advances+in+Neural+Information+Process)
9. Atharva Gundawar, Mudit Verma, Lin Guan, Karthik Valmeekam, Siddhant Bhambri, and Subbarao Kambhampati. 2024. Robust planning with llm-modulo framework: Case study in travel planning. arXiv preprint arXiv:2405.20625. — [arXiv:2405.20625](https://arxiv.org/abs/2405.20625) [find](https://scholar.google.com/scholar?q=Atharva+Gundawar%2C+Mudit+Verma%2C+Lin+Guan%2C+Karthik+Valmeekam%2C+Siddhant+Bhambri%2C+and+Subbarao+Kambhampati%2E+2024%2E+Robust+planning+with+llm%2Dmodulo+framework%3A+Case+study+in+travel+planning%2E+arXiv+preprint+a)
10. Michael Gusenbauer and Neal R Haddaway. 2020. Which academic search systems are suitable for systematic reviews or meta-analyses? evaluating retrieval qualities of google scholar, pubmed, and 26 other resources. Research synthesis methods, 11(2):181–217. [find](https://scholar.google.com/scholar?q=Michael+Gusenbauer+and+Neal+R+Haddaway%2E+2020%2E+Which+academic+search+systems+are+suitable+for+systematic+reviews+or+meta%2Danalyses%3F+evaluating+retrieval+qualities+of+google+scholar%2C+pubmed%2C+and+26+other)
11. Michael Gusenbauer and Neal R Haddaway. 2021. What every researcher should know about searching– clarified concepts, search advice, and an agenda to improve finding in academia. Research synthesis methods, 12(2):136–147. [find](https://scholar.google.com/scholar?q=Michael+Gusenbauer+and+Neal+R+Haddaway%2E+2021%2E+What+every+researcher+should+know+about+searching%E2%80%93+clarified+concepts%2C+search+advice%2C+and+an+agenda+to+improve+finding+in+academia%2E+Research+synthesis+met)
12. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). arXiv preprint arXiv:2410.21276.
13. Karl Kingsley, Gillian M Galbraith, Matthew Herring, Eva Stowers, Tanis Stewart, and Karla V Kingsley. 2011. Why not just google it? an assessment of information literacy skills in a biomedical science curriculum. BMC medical education, 11:1–8. [find](https://scholar.google.com/scholar?q=Karl+Kingsley%2C+Gillian+M+Galbraith%2C+Matthew+Herring%2C+Eva+Stowers%2C+Tanis+Stewart%2C+and+Karla+V+Kingsley%2E+2011%2E+Why+not+just+google+it%3F+an+assessment+of+information+literacy+skills+in+a+biomedical+scienc)
14. Minghan Li, Honglei Zhuang, Kai Hui, Zhen Qin, Jimmy Lin, Rolf Jagerman, Xuanhui Wang, and Michael Bendersky. 2023. Generate, filter, and fuse: Query expansion via multi-step keyword generation for zero-shot neural rankers. arXiv preprint arXiv:2311.09175. — [arXiv:2311.09175](https://arxiv.org/abs/2311.09175) [find](https://scholar.google.com/scholar?q=Minghan+Li%2C+Honglei+Zhuang%2C+Kai+Hui%2C+Zhen+Qin%2C+Jimmy+Lin%2C+Rolf+Jagerman%2C+Xuanhui+Wang%2C+and+Michael+Bendersky%2E+2023%2E+Generate%2C+filter%2C+and+fuse%3A+Query+expansion+via+multi%2Dstep+keyword+generation+for+ze)
15. Zhehui Liao, Maria Antoniak, Inyoung Cheong, Evie Yu-Yen Cheng, Ai-Heng Lee, Kyle Lo, Joseph Chee Chang, and Amy X Zhang. 2024. Llms as research tools: A large scale survey of researchers’ usage and perceptions. arXiv preprint arXiv:2411.05025. — [arXiv:2411.05025](https://arxiv.org/abs/2411.05025) [find](https://scholar.google.com/scholar?q=Zhehui+Liao%2C+Maria+Antoniak%2C+Inyoung+Cheong%2C+Evie+Yu%2DYen+Cheng%2C+Ai%2DHeng+Lee%2C+Kyle+Lo%2C+Joseph+Chee+Chang%2C+and+Amy+X+Zhang%2E+2024%2E+Llms+as+research+tools%3A+A+large+scale+survey+of+researchers%E2%80%99+usage+and+p)
16. Zhihan Liu, Hao Hu, Shenao Zhang, Hongyi Guo, Shuqi Ke, Boyi Liu, and Zhaoran Wang. 2023. Reason for future, act for now: A principled framework for autonomous llm agents with provable sample efficiency. arXiv preprint arXiv:2309.17382. [find](https://scholar.google.com/scholar?q=Zhihan+Liu%2C+Hao+Hu%2C+Shenao+Zhang%2C+Hongyi+Guo%2C+Shuqi+Ke%2C+Boyi+Liu%2C+and+Zhaoran+Wang%2E+2023%2E+Reason+for+future%2C+act+for+now%3A+A+principled+framework+for+autonomous+llm+agents+with+provable+sample+efficien)
17. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. [The ai scientist: Towards fully automated open-ended scientific discovery](https://md.research.elamur.ai/free/en/papers/lu-2024-ai-scientist-fully-automated/). arXiv preprint arXiv:2408.06292.
18. Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. 2024. Augmenting large language models with chemistry tools. Nature Machine Intelligence, pages 1–11. [find](https://scholar.google.com/scholar?q=Andres+M%2E+Bran%2C+Sam+Cox%2C+Oliver+Schilter%2C+Carlo+Baldassari%2C+Andrew+D+White%2C+and+Philippe+Schwaller%2E+2024%2E+Augmenting+large+language+models+with+chemistry+tools%2E+Nature+Machine+Intelligence%2C+pages+1%E2%80%9311)
19. Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query rewriting for retrieval-augmented large language models. arXiv preprint arXiv:2305.14283. — [arXiv:2305.14283](https://arxiv.org/abs/2305.14283) [find](https://scholar.google.com/scholar?q=Xinbei+Ma%2C+Yeyun+Gong%2C+Pengcheng+He%2C+Hai+Zhao%2C+and+Nan+Duan%2E+2023%2E+Query+rewriting+for+retrieval%2Daugmented+large+language+models%2E+arXiv+preprint+arXiv%3A2305%2E14283%2E+%E2%80%94+%5BarXiv%3A2305%2E14283%5D%28https%3A%2F%2Farxiv%2Eor)
20. Lisa Messeri and MJ Crockett. 2024. Artificial intelligence and illusions of understanding in scientific research. Nature, 627(8002):49–58. [find](https://scholar.google.com/scholar?q=Lisa+Messeri+and+MJ+Crockett%2E+2024%2E+Artificial+intelligence+and+illusions+of+understanding+in+scientific+research%2E+Nature%2C+627%288002%29%3A49%E2%80%9358%2E)
21. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
22. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–22. [find](https://scholar.google.com/scholar?q=Joon+Sung+Park%2C+Joseph+O%E2%80%99Brien%2C+Carrie+Jun+Cai%2C+Meredith+Ringel+Morris%2C+Percy+Liang%2C+and+Michael+S+Bernstein%2E+2023%2E+Generative+agents%3A+Interactive+simulacra+of+human+behavior%2E+In+Proceedings+of+the+36)
23. Wenjun Peng, Guiyang Li, Yue Jiang, Zilong Wang, Dan Ou, Xiaoyi Zeng, Derong Xu, Tong Xu, and Enhong Chen. 2024. Large language model based long-tail query rewriting in taobao search. In Companion Proceedings of the ACM on Web Conference 2024, pages 20–28. [find](https://scholar.google.com/scholar?q=Wenjun+Peng%2C+Guiyang+Li%2C+Yue+Jiang%2C+Zilong+Wang%2C+Dan+Ou%2C+Xiaoyi+Zeng%2C+Derong+Xu%2C+Tong+Xu%2C+and+Enhong+Chen%2E+2024%2E+Large+language+model+based+long%2Dtail+query+rewriting+in+taobao+search%2E+In+Companion+Pro)
24. Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. 2024. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.07199. [find](https://scholar.google.com/scholar?q=Pranav+Putta%2C+Edmund+Mills%2C+Naman+Garg%2C+Sumeet+Motwani%2C+Chelsea+Finn%2C+Divyansh+Garg%2C+and+Rafael+Rafailov%2E+2024%2E+Agent+q%3A+Advanced+reasoning+and+learning+for+autonomous+ai+agents%2E+arXiv+preprint+arXiv%3A)
25. Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. 2023. [Communicative agents for software development](https://md.research.elamur.ai/free/en/papers/qian-2023-chatdev-communicative-agents-software/). arXiv preprint arXiv:2307.07924.
26. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+2017%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2E)
27. Richard Van Noorden and Jeffrey M Perkel. 2023. Ai and science: what 1,600 researchers think. Nature, 621(7980):672–675. [find](https://scholar.google.com/scholar?q=Richard+Van+Noorden+and+Jeffrey+M+Perkel%2E+2023%2E+Ai+and+science%3A+what+1%2C600+researchers+think%2E+Nature%2C+621%287980%29%3A672%E2%80%93675%2E)
28. Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2024a. SciMON: Scientific inspiration machines optimized for novelty. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 279–299, Bangkok, Thailand. Association for Computational… [find](https://scholar.google.com/scholar?q=Qingyun+Wang%2C+Doug+Downey%2C+Heng+Ji%2C+and+Tom+Hope%2E+2024a%2E+SciMON%3A+Scientific+inspiration+machines+optimized+for+novelty%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Association+for+Computational+L)
29. Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Min Zhang, Qingsong Wen, et al. 2024b. Autosurvey: Large language models can automatically write surveys. arXiv preprint arXiv:2406.10252. [find](https://scholar.google.com/scholar?q=Yidong+Wang%2C+Qi+Guo%2C+Wenjin+Yao%2C+Hongbo+Zhang%2C+Xin+Zhang%2C+Zhen+Wu%2C+Meishan+Zhang%2C+Xinyu+Dai%2C+Min+Zhang%2C+Qingsong+Wen%2C+et+al%2E+2024b%2E+Autosurvey%3A+Large+language+models+can+automatically+write+surveys%2E+a)
30. Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 1–10. [find](https://scholar.google.com/scholar?q=Frank+F+Xu%2C+Uri+Alon%2C+Graham+Neubig%2C+and+Vincent+Josua+Hellendoorn%2E+2022%2E+A+systematic+evaluation+of+large+language+models+of+code%2E+In+Proceedings+of+the+6th+ACM+SIGPLAN+International+Symposium+on+Mac)
31. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+et+al%2E+2024%2E+Qwen2%2E+5+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E15115%2E)
32. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. [React: synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/) (2022). arXiv preprint arXiv:2210.03629.

*Updated 1 November 2026*