---
title: "AgentBoard — paper by Ma et al. (2024) — summary and ideas"
description: "Most benchmarks for LLM agents judge models only by whether they finish a task. 62 references, cited by 68 papers here."
authors: ["Chang Ma", "Junlei Zhang", "Zhihao Zhu", "Cheng Yang", "Yujiu Yang", "Yaohui Jin", "Zhenzhong Lan", "Lingpeng Kong", "Junxian He"]
year: 2024
venue: "arXiv:2401.13178"
source: "https://arxiv.org/abs/2401.13178"
doi: "10.48550/arxiv.2401.13178"
arxiv: "2401.13178"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ma-2024-agentboard-analytical-evaluation-board-multi/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ma-2024-agentboard-analytical-evaluation-board-multi/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents

# AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents

**Authors:** [Chang Ma](https://md.research.elamur.ai/free/en/authors/chang-ma/), [Junlei Zhang](https://md.research.elamur.ai/free/en/authors/junlei-zhang/), [Zhihao Zhu](https://md.research.elamur.ai/free/en/authors/zhihao-zhu/), [Cheng Yang](https://md.research.elamur.ai/free/en/authors/cheng-hong-yang/), [Yujiu Yang](https://md.research.elamur.ai/free/en/authors/yujiu-yang/), [Yaohui Jin](https://md.research.elamur.ai/free/en/authors/yaohui-jin/), [Zhenzhong Lan](https://md.research.elamur.ai/free/en/authors/zhenzhong-lan/), [Lingpeng Kong](https://md.research.elamur.ai/free/en/authors/lingpeng-kong/) and [Junxian He](https://md.research.elamur.ai/free/en/authors/junxian-he/) · **Year:** 2024 · **Venue:** [arXiv:2401.13178](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2024](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2024/)) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2401.13178) · [PDF](https://arxiv.org/pdf/2401.13178) · [DOI 10.48550/arxiv.2401.13178](https://doi.org/10.48550/arxiv.2401.13178) · [arXiv 2401.13178](https://arxiv.org/abs/2401.13178) · [OpenAlex](https://openalex.org/W4415797149)

**Citations:** 22 citations (OpenAlex, 2026-10-14) · [68](https://md.research.elamur.ai/free/en/papers/ma-2024-agentboard-analytical-evaluation-board-multi/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most benchmarks for LLM agents judge models only by whether they finish a task. This hides differences when nearly every model fails and reveals little about why. Many benchmarks also rely on single-turn or fully observable problems. AgentBoard (NeurIPS 2024, Datasets and Benchmarks track) addresses this with 1013 text-based environments spread over nine tasks in four areas: embodied AI, games, web, and tools. Humans checked each environment to confirm it requires several rounds of interaction under partial observability. Subgoals were defined or hand-annotated for every sample, which allows a progress rate that credits partial advancement. An open-source toolkit with a Wandb-based panel lets users break down results by difficulty, sub-skill, interaction length, action validity (grounding), and full trajectories. The authors used a simple reflex agent that maps the goal plus history to the next action, and tested both proprietary and open-weight models. GPT-4 was best overall, and Llama3 and DeepSeek LLM were the strongest open models. Progress rate exposed gains that success rate missed. Weak multi-turn ability set most open models apart. The resource supports diagnosing agent weaknesses, not just ranking systems.

## Key points
- Tasks: AlfWorld, ScienceWorld, BabyAI (embodied); Jericho, PDDL (game); WebShop, WebArena (web); Tool Query, Tool Operation (tool). Together they contain 1013 environments.
- Interaction is modeled as a POMDP. The baseline agent conditions on the goal and on a memory of past observations and actions.
- Six sub-skills are scored: memory, planning, world modeling, self-reflection, grounding, and spatial navigation. GPT-4 leads on all of them, and open-weight models show uneven gaps.
- The authors cite Ruan et al. (2024), who found that AgentBoard scores correlate with reasoning and coding benchmarks, while AgentBench scores track the knowledge test MMLU.
- Limitations: progress rate depends on human-written subgoals, since current LLMs cannot generate them reliably, and evaluation is restricted to simulated environments. Code and data are available at github.com/hkust-nlp/AgentBoard.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong and Junxian He. 2024. AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents. arXiv:2401.13178.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ma2024agentboard,
  title = {{AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents}},
  author = {Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong and Junxian He},
  year = {2024},
  doi = {10.48550/arxiv.2401.13178},
  eprint = {2401.13178},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.13178}
}
```

- [← previous “My Answer is C”: First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models](https://md.research.elamur.ai/free/en/papers/wang-2024-my-answer-is-c-first/)
- [next → Better & Faster Large Language Models via Multi-token Prediction](https://md.research.elamur.ai/free/en/papers/gloeckle-2024-better-faster-large-language-models/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

62 references, 33 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al. (2022). [Do as i can, not as i say: Grounding language in robotic affordances](https://md.research.elamur.ai/free/en/papers/ahn-2022-i-i-say-grounding-language/). ArXiv preprint, abs/2204.01691.
2. Anthropic (2023). Introducing claude. [find](https://scholar.google.com/scholar?q=Anthropic+%282023%29%2E+Introducing+claude%2E)
3. Chase, H. (2022). Langchain. [find](https://scholar.google.com/scholar?q=Chase%2C+H%2E+%282022%29%2E+Langchain%2E)
4. Chevalier-Boisvert, M., Bahdanau, D., Lahlou, S., Willems, L., Saharia, C., Nguyen, T. H., and Bengio, Y. (2019). [Babyai: A platform to study the sample efficiency of grounded language learning](https://md.research.elamur.ai/free/en/papers/chevalier-boisvert-2018-babyai-platform-study-sample-efficiency/). In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 20…
5. Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., et al. (2023). Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See [https://vicuna.lmsys](https://vicuna.lmsys). org (accessed 14 April 2023). [↗](https://vicuna.lmsys)
6. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
7. DeepSeek-AI, :, Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., Gao, H., Gao, K., Gao, W., Ge, R., Guan, K., Guo, D., Guo, J., Hao, G., Hao, Z., He, Y., Hu, W., Huang, P., Li, E., Li, G., Li, J., Li, Y., Li, Y. K., Liang, W., Lin, F., Liu, A. X., Liu, B.,… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+%3A%2C+Bi%2C+X%2E%2C+Chen%2C+D%2E%2C+Chen%2C+G%2E%2C+Chen%2C+S%2E%2C+Dai%2C+D%2E%2C+Deng%2C+C%2E%2C+Ding%2C+H%2E%2C+Dong%2C+K%2E%2C+Du%2C+Q%2E%2C+Fu%2C+Z%2E%2C+Gao%2C+H%2E%2C+Gao%2C+K%2E%2C+Gao%2C+W%2E%2C+Ge%2C+R%2E%2C+Guan%2C+K%2E%2C+Guo%2C+D%2E%2C+Guo%2C+J%2E%2C+Hao%2C+G%2E%2C+Hao%2C+Z%2E%2C+He%2C+Y%2E%2C+Hu)
8. Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. (2023). [Mind2web: Towards a generalist agent for the web](https://md.research.elamur.ai/free/en/papers/deng-2023-mind2web-towards-generalist-agent-web/). ArXiv preprint, abs/2306.06070.
9. Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. (2023). Palm-e: An embodied multimodal language model. ArXiv preprint, abs/2303.03378. [find](https://scholar.google.com/scholar?q=Driess%2C+D%2E%2C+Xia%2C+F%2E%2C+Sajjadi%2C+M%2E+S%2E%2C+Lynch%2C+C%2E%2C+Chowdhery%2C+A%2E%2C+Ichter%2C+B%2E%2C+Wahid%2C+A%2E%2C+Tompson%2C+J%2E%2C+Vuong%2C+Q%2E%2C+Yu%2C+T%2E%2C+et+al%2E+%282023%29%2E+Palm%2De%3A+An+embodied+multimodal+language+model%2E+ArXiv+preprint%2C+abs%2F)
10. Dziri, N., Lu, X., Sclar, M., Li, X. L., Jian, L., Lin, B. Y., West, P., Bhagavatula, C., Bras, R. L., Hwang, J. D., et al. (2023). [Faith and fate: Limits of transformers on compositionality](https://md.research.elamur.ai/free/en/papers/dziri-2023-faith-fate-limits-transformers-compositionality/). ArXiv preprint, abs/2305.18654.
11. Gu, Y., Deng, X., and Su, Y. (2022). Don’t generate, discriminate: A proposal for grounding language models to real-world environments. ArXiv preprint, abs/2212.09736. [find](https://scholar.google.com/scholar?q=Gu%2C+Y%2E%2C+Deng%2C+X%2E%2C+and+Su%2C+Y%2E+%282022%29%2E+Don%E2%80%99t+generate%2C+discriminate%3A+A+proposal+for+grounding+language+models+to+real%2Dworld+environments%2E+ArXiv+preprint%2C+abs%2F2212%2E09736%2E)
12. Hausknecht, M. J., Ammanabrolu, P., Côté, M., and Yuan, X. (2020). Interactive fiction games: A colossal adventure. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI S… [find](https://scholar.google.com/scholar?q=Hausknecht%2C+M%2E+J%2E%2C+Ammanabrolu%2C+P%2E%2C+C%C3%B4t%C3%A9%2C+M%2E%2C+and+Yuan%2C+X%2E+%282020%29%2E+Interactive+fiction+games%3A+A+colossal+adventure%2E+In+The+Thirty%2DFourth+AAAI+Conference+on+Artificial+Intelligence%2C+AAAI+2020%2C+The+Thir)
13. Hong, S., Zheng, X., Chen, J., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., Zhou, L., et al. (2023). [Metagpt: Meta programming for multi-agent collaborative framework](https://md.research.elamur.ai/free/en/papers/hong-2023-metagpt-meta-programming-multi/). arXiv preprint arXiv:2308.00352.
14. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023). Mistral 7b. ArXiv preprint, abs/2310.06825. [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E%2C+Sablayrolles%2C+A%2E%2C+Mensch%2C+A%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2E+S%2E%2C+Casas%2C+D%2E+d%2E+l%2E%2C+Bressand%2C+F%2E%2C+Lengyel%2C+G%2E%2C+Lample%2C+G%2E%2C+Saulnier%2C+L%2E%2C+et+al%2E+%282023%29%2E+Mistral+7b%2E+ArXiv+preprint%2C+abs%2F2310%2E06825)
15. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361.
16. Kinniment, M., Sato, L. J. K., Du, H., Goodrich, B., Hasin, M., Chan, L., Miles, L. H., Lin, T. R., Wijk, H., Burget, J., et al. (2023). Evaluating language-model agents on realistic autonomous tasks. ArXiv preprint, abs/2312.11671. — [arXiv:2312.11671](https://arxiv.org/abs/2312.11671) [find](https://scholar.google.com/scholar?q=Kinniment%2C+M%2E%2C+Sato%2C+L%2E+J%2E+K%2E%2C+Du%2C+H%2E%2C+Goodrich%2C+B%2E%2C+Hasin%2C+M%2E%2C+Chan%2C+L%2E%2C+Miles%2C+L%2E+H%2E%2C+Lin%2C+T%2E+R%2E%2C+Wijk%2C+H%2E%2C+Burget%2C+J%2E%2C+et+al%2E+%282023%29%2E+Evaluating+language%2Dmodel+agents+on+realistic+autonomous+tasks%2E)
17. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. (2023). [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626.
18. Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Denison, C., Hernandez, D., Li, D., Durmus, E., Hubinger, E., Kernion, J., et al. (2023). [Measuring faithfulness in chain-of-thought reasoning](https://md.research.elamur.ai/free/en/papers/lanham-2023-measuring-faithfulness-chain-thought-reasoning/). ArXiv preprint, abs/2307.13702.
19. LeCun, Y. (2022). A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62. [find](https://scholar.google.com/scholar?q=LeCun%2C+Y%2E+%282022%29%2E+A+path+towards+autonomous+machine+intelligence+version+0%2E9%2E+2%2C+2022%2D06%2D27%2E+Open+Review%2C+62%2E)
20. Li, M., Song, F., Yu, B., Yu, H., Li, Z., Huang, F., and Li, Y. (2023). Api-bank: A benchmark for tool-augmented llms. ArXiv preprint, abs/2304.08244. — [arXiv:2304.08244](https://arxiv.org/abs/2304.08244) [find](https://scholar.google.com/scholar?q=Li%2C+M%2E%2C+Song%2C+F%2E%2C+Yu%2C+B%2E%2C+Yu%2C+H%2E%2C+Li%2C+Z%2E%2C+Huang%2C+F%2E%2C+and+Li%2C+Y%2E+%282023%29%2E+Api%2Dbank%3A+A+benchmark+for+tool%2Daugmented+llms%2E+ArXiv+preprint%2C+abs%2F2304%2E08244%2E+%E2%80%94+%5BarXiv%3A2304%2E08244%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2304%2E0)
21. Lin, Y.-T. and Chen, Y.-N. (2023). [Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models](https://md.research.elamur.ai/free/en/papers/lin-2023-llm-eval-unified-multi-dimensional/). ArXiv preprint, abs/2305.13711.
22. Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., et al. (2023a). [Agentbench: Evaluating llms as agents](https://md.research.elamur.ai/free/en/papers/liu-2023-agentbench-evaluating-llms-agents/). ArXiv preprint, abs/2308.03688.
23. Liu, Z., Yao, W., Zhang, J., Xue, L., Heinecke, S., Murthy, R., Feng, Y., Chen, Z., Niebles, J. C., Arpit, D., et al. (2023b). Bolaa: Benchmarking and orchestrating llm-augmented autonomous agents. ArXiv preprint, abs/2308.05960. [find](https://scholar.google.com/scholar?q=Liu%2C+Z%2E%2C+Yao%2C+W%2E%2C+Zhang%2C+J%2E%2C+Xue%2C+L%2E%2C+Heinecke%2C+S%2E%2C+Murthy%2C+R%2E%2C+Feng%2C+Y%2E%2C+Chen%2C+Z%2E%2C+Niebles%2C+J%2E+C%2E%2C+Arpit%2C+D%2E%2C+et+al%2E+%282023b%29%2E+Bolaa%3A+Benchmarking+and+orchestrating+llm%2Daugmented+autonomous+agents%2E+Ar)
24. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023). Self-refine: Iterative refinement with self-feedback. ArXiv preprint, abs/2303.17651. [find](https://scholar.google.com/scholar?q=Madaan%2C+A%2E%2C+Tandon%2C+N%2E%2C+Gupta%2C+P%2E%2C+Hallinan%2C+S%2E%2C+Gao%2C+L%2E%2C+Wiegreffe%2C+S%2E%2C+Alon%2C+U%2E%2C+Dziri%2C+N%2E%2C+Prabhumoye%2C+S%2E%2C+Yang%2C+Y%2E%2C+et+al%2E+%282023%29%2E+Self%2Drefine%3A+Iterative+refinement+with+self%2Dfeedback%2E+ArXiv+prepr)
25. Mialon, G., Fourrier, C., Swift, C., Wolf, T., LeCun, Y., and Scialom, T. (2023). [Gaia: a benchmark for general ai assistants](https://md.research.elamur.ai/free/en/papers/mialon-2023-gaia-benchmark-general-ai-assistants/). ArXiv preprint, abs/2311.12983.
26. OpenAI (2022). Introducing chatgpt. [find](https://scholar.google.com/scholar?q=OpenAI+%282022%29%2E+Introducing+chatgpt%2E)
27. OpenAI (2023). [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv, pages 2303–08774.
28. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022). [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744.
29. Pourchot, A. and Sigaud, O. (2019). CEM-RL: combining evolutionary and gradient-based methods for policy search. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. [find](https://scholar.google.com/scholar?q=Pourchot%2C+A%2E+and+Sigaud%2C+O%2E+%282019%29%2E+CEM%2DRL%3A+combining+evolutionary+and+gradient%2Dbased+methods+for+policy+search%2E+In+7th+International+Conference+on+Learning+Representations%2C+ICLR+2019%2C+New+Orleans%2C+LA)
30. Puterman, M. L. (1990). Markov decision processes. Handbooks in operations research and management science, 2:331–434. [find](https://scholar.google.com/scholar?q=Puterman%2C+M%2E+L%2E+%281990%29%2E+Markov+decision+processes%2E+Handbooks+in+operations+research+and+management+science%2C+2%3A331%E2%80%93434%2E)
31. Qin, Y., Hu, S., Lin, Y., Chen, W., Ding, N., Cui, G., Zeng, Z., Huang, Y., Xiao, C., Han, C., et al. (2023a). [Tool learning with foundation models](https://md.research.elamur.ai/free/en/papers/qin-2023-tool-learning-foundation-models/). ArXiv preprint, abs/2304.08354.
32. Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., et al. (2023b). [Toolllm: Facilitating large language models to master 16000+ real-world apis](https://md.research.elamur.ai/free/en/papers/qin-2023-toolllm-facilitating-large-language-models/). ArXiv preprint, abs/2307.16789.
33. Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg, J. T., et al. (2022). A generalist agent. ArXiv preprint, abs/2205.06175. [find](https://scholar.google.com/scholar?q=Reed%2C+S%2E%2C+Zolna%2C+K%2E%2C+Parisotto%2C+E%2E%2C+Colmenarejo%2C+S%2E+G%2E%2C+Novikov%2C+A%2E%2C+Barth%2DMaron%2C+G%2E%2C+Gimenez%2C+M%2E%2C+Sulsky%2C+Y%2E%2C+Kay%2C+J%2E%2C+Springenberg%2C+J%2E+T%2E%2C+et+al%2E+%282022%29%2E+A+generalist+agent%2E+ArXiv+preprint%2C+abs%2F2205)
34. Richards, T. B. (2023). Significant-gravitas/autogpt: An experimental open-source attempt to make gpt-4 fully autonomous. [find](https://scholar.google.com/scholar?q=Richards%2C+T%2E+B%2E+%282023%29%2E+Significant%2Dgravitas%2Fautogpt%3A+An+experimental+open%2Dsource+attempt+to+make+gpt%2D4+fully+autonomous%2E)
35. Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al. (2023). [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). ArXiv preprint, abs/2308.12950.
36. Ruan, Y., Maddison, C. J., and Hashimoto, T. (2024). [Observational scaling laws and the predictability of language model performance](https://md.research.elamur.ai/free/en/papers/ruan-2024-observational-scaling-laws-predictability-language/). arXiv preprint arXiv:2405.10938.
37. Russell, S. and Norvig, P. (2005). Ai a modern approach. Learning, 2(3):4. [find](https://scholar.google.com/scholar?q=Russell%2C+S%2E+and+Norvig%2C+P%2E+%282005%29%2E+Ai+a+modern+approach%2E+Learning%2C+2%283%29%3A4%2E)
38. Shi, T., Karpathy, A., Fan, L., Hernandez, J., and Liang, P. (2017a). World of bits: An open-domain platform for web-based agents. In Precup, D. and Teh, Y. W., editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume… [find](https://scholar.google.com/scholar?q=Shi%2C+T%2E%2C+Karpathy%2C+A%2E%2C+Fan%2C+L%2E%2C+Hernandez%2C+J%2E%2C+and+Liang%2C+P%2E+%282017a%29%2E+World+of+bits%3A+An+open%2Ddomain+platform+for+web%2Dbased+agents%2E+In+Precup%2C+D%2E+and+Teh%2C+Y%2E+W%2E%2C+editors%2C+Proceedings+of+the+34th+Intern)
39. Shi, T., Karpathy, A., Fan, L., Hernandez, J., and Liang, P. (2017b). World of bits: An open-domain platform for web-based agents. In Precup, D. and Teh, Y. W., editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume… [find](https://scholar.google.com/scholar?q=Shi%2C+T%2E%2C+Karpathy%2C+A%2E%2C+Fan%2C+L%2E%2C+Hernandez%2C+J%2E%2C+and+Liang%2C+P%2E+%282017b%29%2E+World+of+bits%3A+An+open%2Ddomain+platform+for+web%2Dbased+agents%2E+In+Precup%2C+D%2E+and+Teh%2C+Y%2E+W%2E%2C+editors%2C+Proceedings+of+the+34th+Intern)
40. Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. R., and Yao, S. (2023). [Reflexion: Language agents with verbal reinforcement learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/). In Thirty-seventh Conference on Neural Information Processing Systems.
41. Shridhar, M., Yuan, X., Côté, M., Bisk, Y., Trischler, A., and Hausknecht, M. J. (2021). [Alfworld: Aligning text and embodied environments for interactive learning](https://md.research.elamur.ai/free/en/papers/shridhar-2020-alfworld-aligning-text-embodied-environments/). In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
42. Silver, T. and Chitnis, R. (2020). Pddlgym: Gym environments from pddl problems. ArXiv preprint, abs/2002.06432. — [arXiv:2002.06432](https://arxiv.org/abs/2002.06432) [find](https://scholar.google.com/scholar?q=Silver%2C+T%2E+and+Chitnis%2C+R%2E+%282020%29%2E+Pddlgym%3A+Gym+environments+from+pddl+problems%2E+ArXiv+preprint%2C+abs%2F2002%2E06432%2E+%E2%80%94+%5BarXiv%3A2002%2E06432%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2002%2E06432%29)
43. Song, C. H., Wu, J., Washington, C., Sadler, B. M., Chao, W.-L., and Su, Y. (2023). [Llm-planner: Few-shot grounded planning for embodied agents with large language models](https://md.research.elamur.ai/free/en/papers/song-2022-llm-planner-few-shot-grounded/). In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2998–3009.
44. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023). [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). ArXiv preprint, abs/2302.13971.
45. Vallati, M., Chrpa, L., Grześ, M., McCluskey, T. L., Roberts, M., Sanner, S., et al. (2015). [find](https://scholar.google.com/scholar?q=Vallati%2C+M%2E%2C+Chrpa%2C+L%2E%2C+Grzes%CC%81%2C+M%2E%2C+McCluskey%2C+T%2E+L%2E%2C+Roberts%2C+M%2E%2C+Sanner%2C+S%2E%2C+et+al%2E+%282015%29%2E)
46. The 2014 international planning competition: Progress and trends. Ai Magazine, 36(3):90–98. [find](https://scholar.google.com/scholar?q=The+2014+international+planning+competition%3A+Progress+and+trends%2E+Ai+Magazine%2C+36%283%29%3A90%E2%80%9398%2E)
47. Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. (2023a). Voyager: An open-ended embodied agent with large language models. ArXiv preprint, abs/2305.16291. [find](https://scholar.google.com/scholar?q=Wang%2C+G%2E%2C+Xie%2C+Y%2E%2C+Jiang%2C+Y%2E%2C+Mandlekar%2C+A%2E%2C+Xiao%2C+C%2E%2C+Zhu%2C+Y%2E%2C+Fan%2C+L%2E%2C+and+Anandkumar%2C+A%2E+%282023a%29%2E+Voyager%3A+An+open%2Dended+embodied+agent+with+large+language+models%2E+ArXiv+preprint%2C+abs%2F2305%2E16291%2E)
48. Wang, R., Jansen, P., Côté, M.-A., and Ammanabrolu, P. (2022). ScienceWorld: Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11279–11298, Abu Dhabi, United Arab Emirates. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Wang%2C+R%2E%2C+Jansen%2C+P%2E%2C+C%C3%B4t%C3%A9%2C+M%2E%2DA%2E%2C+and+Ammanabrolu%2C+P%2E+%282022%29%2E+ScienceWorld%3A+Is+your+agent+smarter+than+a+5th+grader%3F+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Language+Pro)
49. Wang, X., Wang, Z., Liu, J., Chen, Y., Yuan, L., Peng, H., and Ji, H. (2023b). [Mint: Evaluating llms in multi-turn interaction with tools and language feedback](https://md.research.elamur.ai/free/en/papers/wang-2023-mint-evaluating-llms-multi-turn/). ArXiv preprint, abs/2309.10691.
50. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022). [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). ArXiv preprint, abs/2206.07682.
51. Wu, Y., Tang, X., Mitchell, T. M., and Li, Y. (2023). Smartplay: A benchmark for llms as intelligent agents. ArXiv preprint, abs/2310.01557. [find](https://scholar.google.com/scholar?q=Wu%2C+Y%2E%2C+Tang%2C+X%2E%2C+Mitchell%2C+T%2E+M%2E%2C+and+Li%2C+Y%2E+%282023%29%2E+Smartplay%3A+A+benchmark+for+llms+as+intelligent+agents%2E+ArXiv+preprint%2C+abs%2F2310%2E01557%2E)
52. Wu, Z., Han, C., Ding, Z., Weng, Z., Liu, Z., Yao, S., Yu, T., and Kong, L. (2024). [Os-copilot: Towards generalist computer agents with self-improvement](https://md.research.elamur.ai/free/en/papers/wu-2024-os-copilot-towards-generalist-computer/). arXiv preprint arXiv:2402.07456.
53. Xie, T., Zhou, F., Cheng, Z., Shi, P., Weng, L., Liu, Y., Hua, T. J., Zhao, J., Liu, Q., Liu, C., et al. (2023). [Openagents: An open platform for language agents in the wild](https://md.research.elamur.ai/free/en/papers/xie-2023-openagents-open-platform-language-agents/). ArXiv preprint, abs/2310.10634.
54. Xu, B., Liu, X., Shen, H., Han, Z., Li, Y., Yue, M., Peng, Z., Liu, Y., Yao, Z., and Xu, D. (2023a). Gentopia: A collaborative platform for tool-augmented llms. arXiv preprint arXiv:2308.04030. [find](https://scholar.google.com/scholar?q=Xu%2C+B%2E%2C+Liu%2C+X%2E%2C+Shen%2C+H%2E%2C+Han%2C+Z%2E%2C+Li%2C+Y%2E%2C+Yue%2C+M%2E%2C+Peng%2C+Z%2E%2C+Liu%2C+Y%2E%2C+Yao%2C+Z%2E%2C+and+Xu%2C+D%2E+%282023a%29%2E+Gentopia%3A+A+collaborative+platform+for+tool%2Daugmented+llms%2E+arXiv+preprint+arXiv%3A2308%2E04030%2E)
55. Xu, Q., Hong, F., Li, B., Hu, C., Chen, Z., and Zhang, J. (2023b). [On the tool manipulation capability of open-source large language models](https://md.research.elamur.ai/free/en/papers/xu-2023-tool-manipulation-capability-open-source/). ArXiv preprint, abs/2305.16504.
56. Xu, Y., Su, H., Xing, C., Mi, B., Liu, Q., Shi, W., Hui, B., Zhou, F., Liu, Y., Xie, T., et al. (2023c). Lemur: Harmonizing natural language and code for language agents. ArXiv preprint, abs/2310.06830. — [arXiv:2310.06830](https://arxiv.org/abs/2310.06830) [find](https://scholar.google.com/scholar?q=Xu%2C+Y%2E%2C+Su%2C+H%2E%2C+Xing%2C+C%2E%2C+Mi%2C+B%2E%2C+Liu%2C+Q%2E%2C+Shi%2C+W%2E%2C+Hui%2C+B%2E%2C+Zhou%2C+F%2E%2C+Liu%2C+Y%2E%2C+Xie%2C+T%2E%2C+et+al%2E+%282023c%29%2E+Lemur%3A+Harmonizing+natural+language+and+code+for+language+agents%2E+ArXiv+preprint%2C+abs%2F2310%2E0683)
57. Yang, Z., Liu, J., Han, Y., Chen, X., Huang, Z., Fu, B., and Yu, G. (2023). [Appagent: Multimodal agents as smartphone users](https://md.research.elamur.ai/free/en/papers/zhang-2023-appagent-multimodal-agents-smartphone-users/). ArXiv preprint, abs/2312.13771.
58. Yao, S., Chen, H., Yang, J., and Narasimhan, K. (2022). [Webshop: Towards scalable real-world web interaction with grounded language agents](https://md.research.elamur.ai/free/en/papers/yao-2022-webshop-towards-scalable-real-world/). Advances in Neural Information Processing Systems, 35:20744–20757.
59. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. (2023). [ReAct: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). In The Eleventh International Conference on Learning Representations.
60. Ye, J., Li, G., Gao, S., Huang, C., Wu, Y., Li, S., Fan, X., Dou, S., Zhang, Q., Gui, T., et al. (2024). [Tooleyes: Fine-grained evaluation for tool learning capabilities of large language models in real-world scenarios](https://md.research.elamur.ai/free/en/papers/ye-2025-tooleyes-fine-grained-evaluation-tool/). ArXiv preprint, abs/2401.00741.
61. Zheng, B., Gou, B., Kil, J., Sun, H., and Su, Y. (2024). [Gpt-4v(ision) is a generalist web agent, if grounded](https://md.research.elamur.ai/free/en/papers/zheng-2024-gpt-4v-ision-generalist-web/). ArXiv preprint, abs/2401.01614.
62. Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Bisk, Y., Fried, D., Alon, U., et al. (2023). [Webarena: A realistic web environment for building autonomous agents](https://md.research.elamur.ai/free/en/papers/zhou-2023-webarena-realistic-web-environment-building/). ArXiv preprint, abs/2307.13854.

*Updated 2 November 2026*