---
title: "TravelPlanner — paper by Xie et al. (2024) — summary and ideas"
description: "The authors introduce TravelPlanner, a benchmark built around multi-day trip planning. 67 references, cited by 111 papers here."
authors: ["Jian Xie", "Kai Zhang", "Jiangjie Chen", "Tinghui Zhu", "Renze Lou", "Yuandong Tian", "Yanghua Xiao", "Yu Su"]
year: 2024
venue: "arXiv:2402.01622"
source: "https://arxiv.org/abs/2402.01622"
doi: "10.48550/arxiv.2402.01622"
arxiv: "2402.01622"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xie-2024-travelplanner-benchmark-real-world-planning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xie-2024-travelplanner-benchmark-real-world-planning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / TravelPlanner: A Benchmark for Real-World Planning with Language Agents

# TravelPlanner: A Benchmark for Real-World Planning with Language Agents

**Authors:** [Jian Xie](https://md.research.elamur.ai/free/en/authors/jian-xie/), [Kai Zhang](https://md.research.elamur.ai/free/en/authors/kai-zhang/), [Jiangjie Chen](https://md.research.elamur.ai/free/en/authors/jiangjie-chen/), [Tinghui Zhu](https://md.research.elamur.ai/free/en/authors/tinghui-zhu/), [Renze Lou](https://md.research.elamur.ai/free/en/authors/renze-lou/), [Yuandong Tian](https://md.research.elamur.ai/free/en/authors/yuandong-tian/), [Yanghua Xiao](https://md.research.elamur.ai/free/en/authors/yanghua-xiao/) and [Yu Su](https://md.research.elamur.ai/free/en/authors/yu-su/) · **Year:** 2024 · **Venue:** arXiv:2402.01622 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.01622) · [PDF](https://arxiv.org/pdf/2402.01622) · [DOI 10.48550/arxiv.2402.01622](https://doi.org/10.48550/arxiv.2402.01622) · [arXiv 2402.01622](https://arxiv.org/abs/2402.01622) · [OpenAlex](https://openalex.org/W4391591592)

**Citations:** 14 citations (OpenAlex, 2026-10-14) · [111](https://md.research.elamur.ai/free/en/papers/xie-2024-travelplanner-benchmark-real-world-planning/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Planning has long been studied in AI, but mostly in narrow settings with a single objective and a fixed correct answer. This paper asks whether LLM-powered language agents can handle open-ended, realistic planning. The authors introduce TravelPlanner, a benchmark built around multi-day trip planning. Agents work in a sandbox of about four million records collected from the web, which they access through six search tools covering cities, flights, distances, restaurants, attractions and lodging. A valid itinerary must meet the user's explicit requirements, such as budget, room type and pet rules, as well as implicit commonsense rules, such as a sensible route and no conflicting transport. Five LLMs, including GPT-4, Gemini and Mixtral, were tested with four planning strategies, including ReAct and Reflexion. The results are poor: GPT-4 satisfies every constraint in only 0.6% of cases, and the other models never do. The benchmark gives the field a demanding, realistic way to measure progress toward human-like planning with many constraints.

## Key points
- The dataset has 1,225 curated queries with reference plans. They fall into nine groups defined by trip length and the number of hard constraints. The splits are 45 training queries with human-annotated plans, 180 validation queries and 1,000 test queries.
- The best final pass rate is 0.6%, achieved by GPT-4. All other evaluated LLMs score zero.
- ReAct and Reflexion fall short on this task. Agents often act against their own reasoning, for example picking expensive options right after noting the need to save money. They also lose track of global constraints such as "Budget" and "Minimum Nights Stay".
- Common failure modes include:
  - wrong tool arguments, such as dates outside the 2022 sandbox data;
  - dead loops that the agent never corrects;
  - hallucinations caused by mixing up retrieved facts, such as reusing one flight number for both legs of a trip.
- Trained human annotators need about 12 minutes per plan, while agents produce one in 1–2 minutes. This suggests real potential for automation once agents become more reliable.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 202 works · 22,509 citations
- [The Ohio State University](https://md.research.elamur.ai/free/en/universities/the-ohio-state-university/) — University, Columbus, United States · 90 works · 19,112 citations
- [Pennsylvania State University](https://md.research.elamur.ai/free/en/universities/pennsylvania-state-university/) — University, State College, United States · 67 works · 5,865 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works

## Cite

Cite as: Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao and Yu Su. 2024. TravelPlanner: A Benchmark for Real-World Planning with Language Agents. arXiv:2402.01622.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{xie2024travelplanner,
  title = {{TravelPlanner: A Benchmark for Real-World Planning with Language Agents}},
  author = {Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao and Yu Su},
  year = {2024},
  doi = {10.48550/arxiv.2402.01622},
  eprint = {2402.01622},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.01622}
}
```

- [← previous Nomic Embed: Training a Reproducible Long Context Text Embedder](https://md.research.elamur.ai/free/en/papers/nussbaum-2024-nomic-embed-training-reproducible-long/)
- [next → Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step](https://md.research.elamur.ai/free/en/papers/zhong-2024-debug-like-human-large-language/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

67 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alterovitz, R., Koenig, S., and Likhachev, M. Robot planning in the real world: Research challenges and opportunities. Ai Magazine, 2016. [find](https://scholar.google.com/scholar?q=Alterovitz%2C+R%2E%2C+Koenig%2C+S%2E%2C+and+Likhachev%2C+M%2E+Robot+planning+in+the+real+world%3A+Research+challenges+and+opportunities%2E+Ai+Magazine%2C+2016%2E)
2. Andreas, J. [Language models as agent models](https://md.research.elamur.ai/free/en/papers/andreas-2022-language-models-as-agent-models/). In Findings of EMNLP, 2022.
3. AutoGPT. Autogpt, 2023. URL [https://github.com/Significant-Gravitas/AutoGPT](https://github.com/Significant-Gravitas/AutoGPT). [↗](https://github.com/Significant-Gravitas/AutoGPT)
4. Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Gianinazzi, L., Gajda, J., Lehmann, T., Podstawski, M., Niewiadomski, H., Nyczyk, P., et al. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXiv:2308.09687, 2023. [find](https://scholar.google.com/scholar?q=Besta%2C+M%2E%2C+Blach%2C+N%2E%2C+Kubicek%2C+A%2E%2C+Gerstenberger%2C+R%2E%2C+Gianinazzi%2C+L%2E%2C+Gajda%2C+J%2E%2C+Lehmann%2C+T%2E%2C+Podstawski%2C+M%2E%2C+Niewiadomski%2C+H%2E%2C+Nyczyk%2C+P%2E%2C+et+al%2E+Graph+of+thoughts%3A+Solving+elaborate+problems+with+la)
5. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Proceedings of NeurIPS, 2020.
6. Campbell, M., Hoane Jr, A. J., and Hsu, F.-h. Deep blue. Artificial intelligence, 2002. [find](https://scholar.google.com/scholar?q=Campbell%2C+M%2E%2C+Hoane+Jr%2C+A%2E+J%2E%2C+and+Hsu%2C+F%2E%2Dh%2E+Deep+blue%2E+Artificial+intelligence%2C+2002%2E)
7. Chen, H., Pasunuru, R., Weston, J., and Celikyilmaz, A. Walking down the memory maze: Beyond context limit through interactive reading. arXiv preprint arXiv:2310.05029, 2023. — [arXiv:2310.05029](https://arxiv.org/abs/2310.05029) [find](https://scholar.google.com/scholar?q=Chen%2C+H%2E%2C+Pasunuru%2C+R%2E%2C+Weston%2C+J%2E%2C+and+Celikyilmaz%2C+A%2E+Walking+down+the+memory+maze%3A+Beyond+context+limit+through+interactive+reading%2E+arXiv+preprint+arXiv%3A2310%2E05029%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E05029%5D%28http)
8. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
9. Cross, S. and Estrada, R. Dart: an example of accelerated evolutionary development. In Proceedings of Workshop on RSP, 1994. [find](https://scholar.google.com/scholar?q=Cross%2C+S%2E+and+Estrada%2C+R%2E+Dart%3A+an+example+of+accelerated+evolutionary+development%2E+In+Proceedings+of+Workshop+on+RSP%2C+1994%2E)
10. Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. [Mind2web: Towards a generalist agent for the web](https://md.research.elamur.ai/free/en/papers/deng-2023-mind2web-towards-generalist-agent-web/). In Proceedings of NeurIPS, 2023.
11. G Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
12. Ge, Y., Hua, W., Ji, J., Tan, J., Xu, S., and Zhang, Y. Openagi: When llm meets domain experts. In Proceedings of NeurIPS, 2023. [find](https://scholar.google.com/scholar?q=Ge%2C+Y%2E%2C+Hua%2C+W%2E%2C+Ji%2C+J%2E%2C+Tan%2C+J%2E%2C+Xu%2C+S%2E%2C+and+Zhang%2C+Y%2E+Openagi%3A+When+llm+meets+domain+experts%2E+In+Proceedings+of+NeurIPS%2C+2023%2E)
13. Georgievski, I. and Aiello, M. Htn planning: Overview, comparison, and beyond. Artificial Intelligence, 2015. [find](https://scholar.google.com/scholar?q=Georgievski%2C+I%2E+and+Aiello%2C+M%2E+Htn+planning%3A+Overview%2C+comparison%2C+and+beyond%2E+Artificial+Intelligence%2C+2015%2E)
14. Grafman, J., Spector, L., and Rattermann, M. J. Planning and the brain. In The cognitive psychology of planning. 2004. [find](https://scholar.google.com/scholar?q=Grafman%2C+J%2E%2C+Spector%2C+L%2E%2C+and+Rattermann%2C+M%2E+J%2E+Planning+and+the+brain%2E+In+The+cognitive+psychology+of+planning%2E+2004%2E)
15. Hayes-Roth, B. and Hayes-Roth, F. A cognitive model of planning. Cognitive science, 1979. [find](https://scholar.google.com/scholar?q=Hayes%2DRoth%2C+B%2E+and+Hayes%2DRoth%2C+F%2E+A+cognitive+model+of+planning%2E+Cognitive+science%2C+1979%2E)
16. Ho, M. K., Saxe, R., and Cushman, F. Planning with theory of mind. Trends in Cognitive Sciences, 2022. [find](https://scholar.google.com/scholar?q=Ho%2C+M%2E+K%2E%2C+Saxe%2C+R%2E%2C+and+Cushman%2C+F%2E+Planning+with+theory+of+mind%2E+Trends+in+Cognitive+Sciences%2C+2022%2E)
17. Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. [Language models as zero-shot planners: Extracting actionable knowledge for embodied agents](https://md.research.elamur.ai/free/en/papers/huang-2022-language-models-zero-shot-planners/). In Proceedings of ICML, 2022.
18. Ichter, B., Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang, E., Julian, R., Kalashnikov, D., Levine, S., Lu, Y., Parada, C., Rao, K., Sermanet, P., Toshev, A., Vanhoucke, V., Xia, F., Xiao, T., Xu, P., Yan, M., Brown, N., Ahn, M., Cortes, O., Sievers,… — [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://md.research.elamur.ai/free/en/papers/ahn-2022-i-i-say-grounding-language/)
19. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E%2C+Sablayrolles%2C+A%2E%2C+Mensch%2C+A%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2E+S%2E%2C+Casas%2C+D%2E+d%2E+l%2E%2C+Bressand%2C+F%2E%2C+Lengyel%2C+G%2E%2C+Lample%2C+G%2E%2C+Saulnier%2C+L%2E%2C+et+al%2E+Mistral+7b%2E+arXiv+preprint+arXiv%3A2310%2E06825%2C+2023%2E)
20. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., La… [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E%2C+Sablayrolles%2C+A%2E%2C+Roux%2C+A%2E%2C+Mensch%2C+A%2E%2C+Savary%2C+B%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2E+S%2E%2C+de+las+Casas%2C+D%2E%2C+Hanna%2C+E%2E+B%2E%2C+Bressand%2C+F%2E%2C+Lengyel%2C+G%2E%2C+Bour%2C+G%2E%2C+Lample%2C+G%2E%2C+Lavaud%2C+L%2E+R%2E%2C+Saulnier%2C)
21. Karpas, E. and Magazzeni, D. Automated planning for robotics. Annual Review of Control, Robotics, and Autonomous Systems, 2020. [find](https://scholar.google.com/scholar?q=Karpas%2C+E%2E+and+Magazzeni%2C+D%2E+Automated+planning+for+robotics%2E+Annual+Review+of+Control%2C+Robotics%2C+and+Autonomous+Systems%2C+2020%2E)
22. Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V. V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., Wu, Y., Neyshabur, B., Gur-Ari, G., and Misra, V. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). In Proceedings of NeurIPS, 2022.
23. Li, M., Zhao, Y., Yu, B., Song, F., Li, H., Yu, H., Li, Z., Huang, F., and Li, Y. [Api-bank: A comprehensive benchmark for tool-augmented llms](https://md.research.elamur.ai/free/en/papers/li-2023-api-bank-comprehensive-benchmark-tool/). In Proceedings of EMNLP, 2023.
24. Liang, X., Wang, B., Huang, H., Wu, S., Wu, P., Lu, L., Ma, Z., and Li, Z. Unleashing infinite-length input capacity for large-scale language models with self-controlled memory system. arXiv preprint arXiv:2304.13343, 2023. — [arXiv:2304.13343](https://arxiv.org/abs/2304.13343) [find](https://scholar.google.com/scholar?q=Liang%2C+X%2E%2C+Wang%2C+B%2E%2C+Huang%2C+H%2E%2C+Wu%2C+S%2E%2C+Wu%2C+P%2E%2C+Lu%2C+L%2E%2C+Ma%2C+Z%2E%2C+and+Li%2C+Z%2E+Unleashing+infinite%2Dlength+input+capacity+for+large%2Dscale+language+models+with+self%2Dcontrolled+memory+system%2E+arXiv+preprint+)
25. Lin, J., Tomlin, N., Andreas, J., and Eisner, J. [Decision-oriented dialogue for human-ai collaboration](https://md.research.elamur.ai/free/en/papers/lin-2024-decision-oriented-dialogue-human-ai/). arXiv preprint arXiv:2305.20076, 2023.
26. Liu, B., Jiang, Y., Zhang, X., Liu, Q., Zhang, S., Biswas, J., and Stone, P. Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.[11477, 2023a](https://md.research.elamur.ai/free/en/papers/liu-2023-llm-p-empowering-large-language/).
27. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). arXiv preprint arXiv:2307.03172, 2023b.
28. Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., et al. [Agentbench: Evaluating llms as agents](https://md.research.elamur.ai/free/en/papers/liu-2023-agentbench-evaluating-llms-agents/). In Proceedings of ICLR, 2024.
29. Lu, P., Peng, B., Cheng, H., Galley, M., Chang, K.-W., Wu, Y. N., Zhu, S.-C., and Gao, J. [Chameleon: Plug-and-play compositional reasoning with large language models](https://md.research.elamur.ai/free/en/papers/lu-2023-chameleon-plug-play-compositional-reasoning/). In Proceedings of NeurIPS, 2023.
30. Mattar, M. G. and Lengyel, M. Planning in the brain. Neuron, 2022. [find](https://scholar.google.com/scholar?q=Mattar%2C+M%2E+G%2E+and+Lengyel%2C+M%2E+Planning+in+the+brain%2E+Neuron%2C+2022%2E)
31. McDermott, D. Robot planning. AI magazine, 1992. [find](https://scholar.google.com/scholar?q=McDermott%2C+D%2E+Robot+planning%2E+AI+magazine%2C+1992%2E)
32. Nakajima, Y. Task-driven autonomous agent utilizing gpt-4, pinecone, and langchain for diverse applications, 2023. [find](https://scholar.google.com/scholar?q=Nakajima%2C+Y%2E+Task%2Ddriven+autonomous+agent+utilizing+gpt%2D4%2C+pinecone%2C+and+langchain+for+diverse+applications%2C+2023%2E)
33. Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. [Webgpt: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/). arXiv preprint arXiv:2112.09332, 2021.
34. OpenAI. Chatgpt, 2022. URL [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt). [↗](https://openai.com/blog/chatgpt)
35. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv:2303.08774, 2023. arXiv preprint Park, J. S., O’Brien, J., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior. In Proceedings of UIST, 2023.
36. Patel, A., Bhattamishra, S., and Goyal, N. [Are NLP models really able to solve simple math word problems?](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/) In Proceedings of NAACL, 2021.
37. Patil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E. [Gorilla: Large language model connected with massive apis](https://md.research.elamur.ai/free/en/papers/patil-2023-gorilla-large-language-model-connected/). arXiv preprint arXiv:2305.15334, 2023.
38. Pinedo, M. Planning and scheduling in manufacturing and services. Springer, 2005. [find](https://scholar.google.com/scholar?q=Pinedo%2C+M%2E+Planning+and+scheduling+in+manufacturing+and+services%2E+Springer%2C+2005%2E)
39. Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., et al. [Toolllm: Facilitating large language models to master 16000+ real-world apis](https://md.research.elamur.ai/free/en/papers/qin-2023-toolllm-facilitating-large-language-models/). In Proceedings of ICLR, 2024.
40. Roy, S. and Roth, D. [Solving general arithmetic word problems](https://md.research.elamur.ai/free/en/papers/roy-2015-solving-general-arithmetic-word-problems/). In Proceedings of EMNLP, 2015.
41. Russell, S. J. and Norvig, P. Artificial intelligence a modern approach. 2010. [find](https://scholar.google.com/scholar?q=Russell%2C+S%2E+J%2E+and+Norvig%2C+P%2E+Artificial+intelligence+a+modern+approach%2E+2010%2E)
42. Schick, T., Dwivedi-Yu, J., Dessi, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T. [Toolformer: Language models can teach themselves to use tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/). In Proceedings of NeurIPS, 2023.
43. Shen, Y., Song, K., Tan, X., Li, D., Lu, W., and Zhuang, Y. [HuggingGPT: Solving AI tasks with chatGPT and its friends in hugging face](https://md.research.elamur.ai/free/en/papers/shen-2023-hugginggpt-solving-ai-tasks-chatgpt/). In Proceedings of NeurIPS, 2023.
44. Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. R., and Yao, S. [Reflexion: Language agents with verbal reinforcement learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/). In Proceedings of NeurIPS, 2023.
45. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. nature, 2016. [find](https://scholar.google.com/scholar?q=Silver%2C+D%2E%2C+Huang%2C+A%2E%2C+Maddison%2C+C%2E+J%2E%2C+Guez%2C+A%2E%2C+Sifre%2C+L%2E%2C+Van+Den+Driessche%2C+G%2E%2C+Schrittwieser%2C+J%2E%2C+Antonoglou%2C+I%2E%2C+Panneershelvam%2C+V%2E%2C+Lanctot%2C+M%2E%2C+et+al%2E+Mastering+the+game+of+go+with+deep+neural)
46. Silver, D., Hasselt, H., Hessel, M., Schaul, T., Guez, A., Harley, T., Dulac-Arnold, G., Reichert, D., Rabinowitz, N., Barreto, A., et al. The predictron: End-to-end learning and planning. In Proceedings of ICML, 2017. [find](https://scholar.google.com/scholar?q=Silver%2C+D%2E%2C+Hasselt%2C+H%2E%2C+Hessel%2C+M%2E%2C+Schaul%2C+T%2E%2C+Guez%2C+A%2E%2C+Harley%2C+T%2E%2C+Dulac%2DArnold%2C+G%2E%2C+Reichert%2C+D%2E%2C+Rabinowitz%2C+N%2E%2C+Barreto%2C+A%2E%2C+et+al%2E+The+predictron%3A+End%2Dto%2Dend+learning+and+planning%2E+In+Proceedi)
47. Song, C. H., Wu, J., Washington, C., Sadler, B. M., Chao, W.-L., and Su, Y. [Llm-planner: Few-shot grounded planning for embodied agents with large language models](https://md.research.elamur.ai/free/en/papers/song-2022-llm-planner-few-shot-grounded/). In Proceedings of ICCV, 2023.
48. Su, Y. Language agents: a critical evolutionary step of artificial intelligence. 2023. URL [https://yusu.substack.com/p/language-agents](https://yusu.substack.com/p/language-agents). [↗](https://yusu.substack.com/p/language-agents)
49. Sumers, T. R., Yao, S., Narasimhan, K., and Griffiths, T. L. Cognitive architectures for language agents. arXiv preprint arXiv:2309.02427, 2023. [find](https://scholar.google.com/scholar?q=Sumers%2C+T%2E+R%2E%2C+Yao%2C+S%2E%2C+Narasimhan%2C+K%2E%2C+and+Griffiths%2C+T%2E+L%2E+Cognitive+architectures+for+language+agents%2E+arXiv+preprint+arXiv%3A2309%2E02427%2C+2023%2E)
50. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023a.
51. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023b.
52. Valmeekam, K., Olmo, A., Sreedharan, S., and Kambhampati, S. [Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change](https://md.research.elamur.ai/free/en/papers/valmeekam-2022-planbench-extensible-benchmark-evaluating-large/). In Proceedings of NeurIPS, 2023.
53. Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023. [find](https://scholar.google.com/scholar?q=Wang%2C+G%2E%2C+Xie%2C+Y%2E%2C+Jiang%2C+Y%2E%2C+Mandlekar%2C+A%2E%2C+Xiao%2C+C%2E%2C+Zhu%2C+Y%2E%2C+Fan%2C+L%2E%2C+and+Anandkumar%2C+A%2E+Voyager%3A+An+open%2Dended+embodied+agent+with+large+language+models%2E+arXiv+preprint+arXiv%3A2305%2E16291%2C+2023%2E)
54. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Proceedings of NeurIPS, 2022.
55. Weng, L. Llm-powered autonomous agents. lilian-weng.github.io, Jun 2023. URL [https://lilianweng.github.io/posts/2023-06-23-agent/](https://lilianweng.github.io/posts/2023-06-23-agent/). [↗](https://lilianweng.github.io/posts/2023-06-23-agent/)
56. Xie, T., Zhou, F., Cheng, Z., Shi, P., Weng, L., Liu, Y., Hua, T. J., Zhao, J., Liu, Q., Liu, C., Liu, L. Z., Xu, Y., Su, H., Shin, D., Xiong, C., and Yu, T. [Openagents: An open platform for language agents in the wild](https://md.research.elamur.ai/free/en/papers/xie-2023-openagents-open-platform-language-agents/). arXiv preprint arXiv:2310.10634, 2023.
57. Xu, Q., Hong, F., Li, B., Hu, C., Chen, Z., and Zhang, J. [On the tool manipulation capability of open-source large language models](https://md.research.elamur.ai/free/en/papers/xu-2023-tool-manipulation-capability-open-source/). arXiv preprint arXiv:2305.16504, 2023.
58. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. [React: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). In Proceedings of ICLR, 2022.
59. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. R. Tree of thoughts: Deliberate problem solving with large language models. In Proceedings of NeurIPS, 2023. [find](https://scholar.google.com/scholar?q=Yao%2C+S%2E%2C+Yu%2C+D%2E%2C+Zhao%2C+J%2E%2C+Shafran%2C+I%2E%2C+Griffiths%2C+T%2E+L%2E%2C+Cao%2C+Y%2E%2C+and+Narasimhan%2C+K%2E+R%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+In+Proceedings+of+NeurIPS%2C+2023%2E)
60. Yuan, S., Chen, J., Fu, Z., Ge, X., Shah, S., Jankowski, C., Xiao, Y., and Yang, D. Distilling script knowledge from large language models for constrained language planning. In Proceedings of ACL, 2023. [find](https://scholar.google.com/scholar?q=Yuan%2C+S%2E%2C+Chen%2C+J%2E%2C+Fu%2C+Z%2E%2C+Ge%2C+X%2E%2C+Shah%2C+S%2E%2C+Jankowski%2C+C%2E%2C+Xiao%2C+Y%2E%2C+and+Yang%2C+D%2E+Distilling+script+knowledge+from+large+language+models+for+constrained+language+planning%2E+In+Proceedings+of+ACL%2C+202)
61. Zhang, Y., Yuan, S., Hu, C., Richardson, K., Xiao, Y., and Chen, J. Timearena: Shaping efficient multitasking language agents in a time-aware simulation. arXiv preprint arXiv:2402.05733, 2024. [find](https://scholar.google.com/scholar?q=Zhang%2C+Y%2E%2C+Yuan%2C+S%2E%2C+Hu%2C+C%2E%2C+Richardson%2C+K%2E%2C+Xiao%2C+Y%2E%2C+and+Chen%2C+J%2E+Timearena%3A+Shaping+efficient+multitasking+language+agents+in+a+time%2Daware+simulation%2E+arXiv+preprint+arXiv%3A2402%2E05733%2C+2024%2E)
62. Zheng, B., Gou, B., Kil, J., Sun, H., and Su, Y. [Gpt-4v(ision) is a generalist web agent, if grounded](https://md.research.elamur.ai/free/en/papers/zheng-2024-gpt-4v-ision-generalist-web/). arXiv preprint arXiv:2401.01614, 2024.
63. Zhong, W., Guo, L., Gao, Q., and Wang, Y. Memorybank: Enhancing large language models with long-term memory. arXiv preprint arXiv:2305.10250, 2023. [find](https://scholar.google.com/scholar?q=Zhong%2C+W%2E%2C+Guo%2C+L%2E%2C+Gao%2C+Q%2E%2C+and+Wang%2C+Y%2E+Memorybank%3A+Enhancing+large+language+models+with+long%2Dterm+memory%2E+arXiv+preprint+arXiv%3A2305%2E10250%2C+2023%2E)
64. Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Bisk, Y., Fried, D., Alon, U., et al. [Webarena: A realistic web environment for building autonomous agents](https://md.research.elamur.ai/free/en/papers/zhou-2023-webarena-realistic-web-environment-building/). In Proceedings of ICLR, 2024.
65. Zhou, W., Jiang, Y. E., Cui, P., Wang, T., Xiao, Z., Hou, Y., Cotterell, R., and Sachan, M. [Recurrentgpt: Interactive generation of (arbitrarily) long text](https://md.research.elamur.ai/free/en/papers/zhou-2023-recurrentgpt-interactive-generation-arbitrarily-long/). arXiv preprint arXiv:2305.13304, 2023.
66. Zhuang, Y., Yu, Y., Wang, K., Sun, H., and Zhang, C. [ToolQA: A dataset for LLM question answering with external tools](https://md.research.elamur.ai/free/en/papers/zhuang-2023-toolqa-dataset-llm-question-answering/). In Proceedings of NeurIPS, 2023.
67. Zhuang, Y., Chen, X., Yu, T., Mitra, S., Bursztyn, V., Rossi, R. A., Sarkhel, S., and Zhang, C. Toolchain*: Efficient action space navigation in large language models with a* search. In Proceedings of ICLR, 2024. [find](https://scholar.google.com/scholar?q=Zhuang%2C+Y%2E%2C+Chen%2C+X%2E%2C+Yu%2C+T%2E%2C+Mitra%2C+S%2E%2C+Bursztyn%2C+V%2E%2C+Rossi%2C+R%2E+A%2E%2C+Sarkhel%2C+S%2E%2C+and+Zhang%2C+C%2E+Toolchain%2A%3A+Efficient+action+space+navigation+in+large+language+models+with+a%2A+search%2E+In+Proceedings+of)

*Updated 2 November 2026*