---
title: "TheAgentCompany — paper by Xu et al. (2024)"
description: "These findings offer useful evidence for both AI adoption decisions and labor-market policy. 39 references, cited by 52 papers here."
authors: ["Frank F. Xu", "Yufan Song", "Boxuan Li", "Yuxuan Tang", "Kritanjali Jain", "Mengxue Bao", "Zora Z. Wang", "Xuhui Zhou", "Zhitong Guo", "Murong Cao", "Mingyang Yang", "Hao Yang Lu", "Amaad Martin", "Zhe Su", "Leander Maben", "Raj Mehta", "Wayne Chi", "Lawrence Jang", "Yiqing Xie", "Shuyan Zhou", "Graham Neubig"]
year: 2024
venue: "arXiv:2412.14161"
source: "https://arxiv.org/abs/2412.14161"
doi: "10.48550/arxiv.2412.14161"
arxiv: "2412.14161"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2024-theagentcompany-benchmarking-llm-agents/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2024-theagentcompany-benchmarking-llm-agents/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks

# TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks

**Authors:** [Frank F. Xu](https://md.research.elamur.ai/free/en/authors/frank-f-xu/), [Yufan Song](https://md.research.elamur.ai/free/en/authors/yufan-song/), [Boxuan Li](https://md.research.elamur.ai/free/en/authors/boxuan-li/), [Yuxuan Tang](https://md.research.elamur.ai/free/en/authors/yuxuan-tang/), Kritanjali Jain, Mengxue Bao, Zora Z. Wang, [Xuhui Zhou](https://md.research.elamur.ai/free/en/authors/xuhui-zhou/), Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, [Zhe Su](https://md.research.elamur.ai/free/en/authors/zhe-su/), Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, [Shuyan Zhou](https://md.research.elamur.ai/free/en/authors/shuyan-zhou/) and [Graham Neubig](https://md.research.elamur.ai/free/en/authors/graham-neubig/) · **Year:** 2024 · **Venue:** [arXiv:2412.14161](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2024](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2024/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2412.14161) · [PDF](https://arxiv.org/pdf/2412.14161) · [DOI 10.48550/arxiv.2412.14161](https://doi.org/10.48550/arxiv.2412.14161) · [arXiv 2412.14161](https://arxiv.org/abs/2412.14161) · [OpenAlex](https://openalex.org/W4405627213)

**Citations:** 16 citations (OpenAlex, 2026-10-14) · [52](https://md.research.elamur.ai/free/en/papers/xu-2024-theagentcompany-benchmarking-llm-agents/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Can LLM-driven agents actually take over real office work, and how much of it? Existing agent benchmarks are often detached from professional practice or limited to one job type, and they rarely test communication with coworkers. TheAgentCompany addresses this by simulating a small software firm inside a fully self-hosted, resettable environment built from open-source tools and filled with realistic mock data. Agents operate from a sandboxed Docker workspace with a browser, code editor and terminal. They must complete 175 tasks spanning engineering, project management, HR, finance, data science and administration, sometimes by chatting with LLM-simulated colleagues. Scoring is checkpoint-based, so partially completed tasks earn partial credit. The authors tested twelve model backbones, both proprietary and open-weights, mainly within the OpenHands framework. The strongest, Gemini 2.5 Pro, fully finished about 30% of tasks. Agents can handle many simpler, well-scoped duties. Long multi-step work, social interaction and complex web UIs remain difficult. These findings offer useful evidence for both AI adoption decisions and labor-market policy.

## Key points
- The intranet runs on self-hostable open-source services: GitLab (code and wiki), OwnCloud (documents), Plane (issues and sprints) and RocketChat (messaging). A pre-packaged VM image is also provided for reproducibility.
- Simulated coworkers are built on the Sotopia platform, each with a profile covering name, role and responsibilities. They are powered by Claude-3-5-Sonnet-20241022.
- Gemini 2.5 Pro performed best, with 30.3% full task completion and a 39.3% partial-credit score. Baselines used OpenHands, and OWL RolePlay served as a second agent scaffold.
- Coding-oriented tasks proved easier than administrative and financial ones. The authors attribute this to abundant public software data and benchmarks, while business data typically stays private.
- Recurring failures include:
  - weak social follow-up, such as not contacting the colleague the agent was referred to;
  - getting stuck on web UI elements, such as popups;
  - "self-deceiving" shortcuts, such as renaming a user to impersonate the intended contact.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 598 works · 191,226 citations
- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 378 works · 161,630 citations
- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 214 works · 241,492 citations
- [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) — Company, Hangzhou, China · 201 works · 17,173 citations
- [University of Hong Kong](https://md.research.elamur.ai/free/en/universities/university-of-hong-kong/) — University, Pok Fu Lam, China · 118 works · 13,496 citations
- [Meta (United Kingdom)](https://md.research.elamur.ai/free/en/companies/meta-united-kingdom/) — Company, London, United Kingdom · 54 works · 19,348 citations
- [Oracle](https://md.research.elamur.ai/free/en/companies/oracle/) — Company, Austin, United States · 9 works · 1,235 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou and Graham Neubig. 2024. TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks. arXiv:2412.14161.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{xu2024theagentcompany,
  title = {{TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks}},
  author = {Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou and Graham Neubig},
  year = {2024},
  doi = {10.48550/arxiv.2412.14161},
  eprint = {2412.14161},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.14161}
}
```

- [← previous Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration](https://md.research.elamur.ai/free/en/papers/feng-2024-modular-pluralism-pluralistic-alignment-multi/)
- [next → VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation](https://md.research.elamur.ai/free/en/papers/ku-2024-viescore-explainable-metrics-conditional-image/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

39 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amodei, D. and Fridman, L. Dario Amodei: Anthropic CEO on Claude, AGI & the Future of AI & Humanity \| Lex Fridman Podcast \#452, November 2024. URL [https://lexfridman.com/](https://lexfridman.com/) dario-amodei-transcript. [↗](https://lexfridman.com/)
2. Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2023. URL [https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf](https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf). [↗](https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf)
3. Bonatti, R., Zhao, D., Bonacci, F., Dupont, D., Abdali, S., Li, Y., Lu, Y., Wagle, J., Koishida, K., Bucker, A., et al. Windows agent arena: Evaluating multi-modal os agents at scale. arXiv preprint arXiv:2409.08264, 2024. — [arXiv:2409.08264](https://arxiv.org/abs/2409.08264) [find](https://scholar.google.com/scholar?q=Bonatti%2C+R%2E%2C+Zhao%2C+D%2E%2C+Bonacci%2C+F%2E%2C+Dupont%2C+D%2E%2C+Abdali%2C+S%2E%2C+Li%2C+Y%2E%2C+Lu%2C+Y%2E%2C+Wagle%2C+J%2E%2C+Koishida%2C+K%2E%2C+Bucker%2C+A%2E%2C+et+al%2E+Windows+agent+arena%3A+Evaluating+multi%2Dmodal+os+agents+at+scale%2E+arXiv+preprint+a)
4. Chezelles, D., Le Sellier, T., Gasse, M., Lacoste, A., Drouin, A., Caccia, M., Boisvert, L., Thakkar, M., Marty, T., Assouel, R., et al. The browsergym ecosystem for web agent research. arXiv preprint arXiv:2412.05467, 2024. [find](https://scholar.google.com/scholar?q=Chezelles%2C+D%2E%2C+Le+Sellier%2C+T%2E%2C+Gasse%2C+M%2E%2C+Lacoste%2C+A%2E%2C+Drouin%2C+A%2E%2C+Caccia%2C+M%2E%2C+Boisvert%2C+L%2E%2C+Thakkar%2C+M%2E%2C+Marty%2C+T%2E%2C+Assouel%2C+R%2E%2C+et+al%2E+The+browsergym+ecosystem+for+web+agent+research%2E+arXiv+preprint)
5. Chollet, F., Knoop, M., Kamradt, G., and Landers, B. ARC Prize 2024: Technical Report, December 2024. URL [https://arcprize.org/media/arc-prize-2024-technical-report.pdf](https://arcprize.org/media/arc-prize-2024-technical-report.pdf). [↗](https://arcprize.org/media/arc-prize-2024-technical-report.pdf)
6. Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. [Mind2web: Towards a generalist agent for the web](https://md.research.elamur.ai/free/en/papers/deng-2023-mind2web-towards-generalist-agent-web/). In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=kiYqbO3wqw. [↗](https://openreview.net/forum?)
7. Drouin, A., Gasse, M., Caccia, M., Laradji, I. H., Verme, M. D., Marty, T., Boisvert, L., Thakkar, M., Cappart, Q., Vazquez, D., Chapados, N., and Lacoste, A. Workarena: How capable are web agents at solving common knowledge work tasks?, 2024. [find](https://scholar.google.com/scholar?q=Drouin%2C+A%2E%2C+Gasse%2C+M%2E%2C+Caccia%2C+M%2E%2C+Laradji%2C+I%2E+H%2E%2C+Verme%2C+M%2E+D%2E%2C+Marty%2C+T%2E%2C+Boisvert%2C+L%2E%2C+Thakkar%2C+M%2E%2C+Cappart%2C+Q%2E%2C+Vazquez%2C+D%2E%2C+Chapados%2C+N%2E%2C+and+Lacoste%2C+A%2E+Workarena%3A+How+capable+are+web+agents+at+)
8. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783, 2024.
9. Eloundou, T., Manning, S., Mishkin, P., and Rock, D. GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models, 2023. [find](https://scholar.google.com/scholar?q=Eloundou%2C+T%2E%2C+Manning%2C+S%2E%2C+Mishkin%2C+P%2E%2C+and+Rock%2C+D%2E+GPTs+are+GPTs%3A+An+Early+Look+at+the+Labor+Market+Impact+Potential+of+Large+Language+Models%2C+2023%2E)
10. He, H., Yao, W., Ma, K., Yu, W., Dai, Y., Zhang, H., Lan, Z., and Yu, D. [Webvoyager: Building an end-to-end web agent with large multimodal models](https://md.research.elamur.ai/free/en/papers/he-2024-webvoyager-building-end-end-web/). arXiv preprint arXiv:2401.13919, 2024.
11. Hu, M., Zhou, Y., Fan, W., Nie, Y., Xia, B., Sun, T., Ye, Z., Jin, Z., Li, Y., Zhang, Z., Wang, Y., Ye, Q., Luo, P., and Li, G. Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation, 2025. URL [https://github.com/camel-ai/owl](https://github.com/camel-ai/owl). [↗](https://github.com/camel-ai/owl)
12. Huang, K.-H., Prabhakar, A., Dhawan, S., Mao, Y., Wang, H., Savarese, S., Xiong, C., Laban, P., and Wu, C.-S. [Crmarena: Understanding the capacity of llm agents to perform professional crm tasks in realistic environments](https://md.research.elamur.ai/free/en/papers/huang-2025-crmarena-understanding-capacity-llm-agents/). arXiv preprint arXiv:2411.02305, 2024.
13. Intelligence, A. A. G. The amazon nova family of models: Technical report and model card. Amazon Technical Reports, 2024. URL [https://www.amazon.science/publications/](https://www.amazon.science/publications/) the-amazon-nova-family-of-models-technical-report-and-model-card. [↗](https://www.amazon.science/publications/)
14. IPython. Jupyter and the future of IPython — IPython. URL [https://ipython.org](https://ipython.org). Jang, L., Li, Y., Ding, C., Lin, J., Liang, P. P., Zhao, D., Bonatti, R., and Koishida, K. Videowebarena: Evaluating long context multimodal agents with video understanding web tasks, 2024. URL https://arxiv.org/abs/2410.… — [arXiv:2410.19100](https://arxiv.org/abs/2410.19100) [↗](https://ipython.org)
15. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. R. [SWE-bench: Can Language Models Resolve Real-world Github Issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=VTF8yNQM66](https://openreview.net/forum?id=VTF8yNQM66). [↗](https://openreview.net/forum?id=VTF8yNQM66)
16. Kambhampati, S., Valmeekam, K., Guan, L., Verma, M., Stechly, K., Bhambri, S., Saldyt, L., and Murthy, A. Llms can’t plan, but can help planning in llm-modulo frameworks. arXiv preprint arXiv:2402.[01817, 2024](https://md.research.elamur.ai/free/en/papers/kambhampati-2024-llms-t-plan-but-help/).
17. Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., and Fried, D. [VisualWebArena: Evaluating multimodal agents on realistic visual web tasks](https://md.research.elamur.ai/free/en/papers/koh-2024-visualwebarena-evaluating-multimodal-agents-realistic/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo…
18. Li, B., Wu, W., Tang, Z., Shi, L., Yang, J., Li, J., Yao, S., Qian, C., Hui, B., Zhang, Q., et al. Devbench: A comprehensive benchmark for software development. arXiv preprint arXiv:2403.08604, 2024. — [arXiv:2403.08604](https://arxiv.org/abs/2403.08604) [find](https://scholar.google.com/scholar?q=Li%2C+B%2E%2C+Wu%2C+W%2E%2C+Tang%2C+Z%2E%2C+Shi%2C+L%2E%2C+Yang%2C+J%2E%2C+Li%2C+J%2E%2C+Yao%2C+S%2E%2C+Qian%2C+C%2E%2C+Hui%2C+B%2E%2C+Zhang%2C+Q%2E%2C+et+al%2E+Devbench%3A+A+comprehensive+benchmark+for+software+development%2E+arXiv+preprint+arXiv%3A2403%2E08604%2C+2024%2E+)
19. Liu, E. Z., Guu, K., Pasupat, P., Shi, T., and Liang, P. [Reinforcement learning on web interfaces using workflow-guided exploration](https://md.research.elamur.ai/free/en/papers/liu-2018-reinforcement-learning-web-interfaces-using/). In International Conference on Learning Representations (ICLR), 2018. URL https://arxiv.org/abs/1802.08802. [↗](https://arxiv.org/abs/1802.08802)
20. Lù, X. H., Kasner, Z., and Reddy, S. [Weblinx: Real-world website navigation with multi-turn dialogue](https://md.research.elamur.ai/free/en/papers/lu-2024-weblinx-real-world-website-navigation/), 2024. URL https://arxiv.org/abs/2402.05930. [↗](https://arxiv.org/abs/2402.05930)
21. Mozilla. Accessibility tree - MDN Web Docs Glossary: Definitions of Web-related terms \| MDN. URL [https://developer.mozilla.org/en-US/docs/Glossary/Accessibility_tree](https://developer.mozilla.org/en-US/docs/Glossary/Accessibility_tree). O*NET. The 29.1 release of the O*NET database, November 2024. URL https://www.onetcenter. org/dictionary/29.1/excel/. [↗](https://developer.mozilla.org/en-US/docs/Glossary/Accessibility_tree)
22. OpenAI. Introducing gpt-4o and more tools to chatgpt free users, 2024. URL [https://openai.com/](https://openai.com/) index/gpt-4o-and-more-tools-to-chatgpt-free/. [↗](https://openai.com/)
23. Park, J. S., O’Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior, 2023. [find](https://scholar.google.com/scholar?q=Park%2C+J%2E+S%2E%2C+O%E2%80%99Brien%2C+J%2E+C%2E%2C+Cai%2C+C%2E+J%2E%2C+Morris%2C+M%2E+R%2E%2C+Liang%2C+P%2E%2C+and+Bernstein%2C+M%2E+S%2E+Generative+agents%3A+Interactive+simulacra+of+human+behavior%2C+2023%2E)
24. Patil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E. [Gorilla: Large language model connected with massive apis](https://md.research.elamur.ai/free/en/papers/patil-2023-gorilla-large-language-model-connected/). arXiv preprint arXiv:2305.15334, 2023.
25. Playwright. Fast and reliable end-to-end testing for modern web apps \| Playwright. URL [https://playwright.dev/](https://playwright.dev/). Rounds, J., Smith, T., Hubert, L., Lewis, P., and Rivkin, D. Development of occupational interest profiles for o* net. Raleigh, NC: National Center for O* NET Development, 1999. [↗](https://playwright.dev/)
26. ServiceNow. BrowserGym: a Gym Environment for Web Task Automation. URL [https://github.com/](https://github.com/) ServiceNow/BrowserGym. Song, Y., Xu, F., Zhou, S., and Neubig, G. Beyond browsing: Api-based web agents. arXiv preprint arXiv:2410.16464, 2024. [↗](https://github.com/)
27. Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
28. Trivedi, H., Khot, T., Hartmann, M., Manku, R., Dong, V., Li, E., Gupta, S., Sabharwal, A., and Balasubramanian, N. [AppWorld: A controllable world of apps and people for benchmarking interactive coding agents](https://md.research.elamur.ai/free/en/papers/trivedi-2024-appworld-controllable-world-apps-people/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics…
29. Wang, X., Chen, Y., Yuan, L., Zhang, Y., Li, Y., Peng, H., and Ji, H. [Executable Code Actions Elicit Better LLM Agents](https://md.research.elamur.ai/free/en/papers/wang-2024-executable-code-actions-elicit-better/). In ICML, 2024a.
30. Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., Tran, H. H., Li, F., Ma, R., Zheng, M., Qian, B., Shao, Y., Muennighoff, N., Zhang, Y., Hui, B., Lin, J., Brennan, R., Peng, H., Ji, H., and Neubig, G. Openhands: An open platform for ai software develo… — [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://md.research.elamur.ai/free/en/papers/wang-2024-openhands-open-platform-ai/)
31. Wittenstein, J. AI Can Only Do 5% of Jobs, Says MIT Economist Who Fears Crash, October 2024. URL [https://www.bloomberg.com/news/articles/2024-10-02/](https://www.bloomberg.com/news/articles/2024-10-02/) ai-can-only-do-5-of-jobs-says-mit-economist-who-fears-crash. [↗](https://www.bloomberg.com/news/articles/2024-10-02/)
32. Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y., Zhou, S., Savarese, S., Xiong, C., Zhong, V., and Yu, T. [OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments](https://md.research.elamur.ai/free/en/papers/xie-2024-osworld-benchmarking-multimodal-agents-open/). In The Thirty-eight Conference on N…
33. Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. [find](https://scholar.google.com/scholar?q=Yang%2C+A%2E%2C+Yang%2C+B%2E%2C+Hui%2C+B%2E%2C+Zheng%2C+B%2E%2C+Yu%2C+B%2E%2C+Zhou%2C+C%2E%2C+Li%2C+C%2E%2C+Li%2C+C%2E%2C+Liu%2C+D%2E%2C+Huang%2C+F%2E%2C+et+al%2E+Qwen2+technical+report%2E+arXiv+preprint+arXiv%3A2407%2E10671%2C+2024%2E)
34. Yang, J., Zhang, H., Li, F., Zou, X., Li, C., and Gao, J. [Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v](https://md.research.elamur.ai/free/en/papers/yang-2023-set-mark-prompting-unleashes-extraordinary/). arXiv preprint arXiv:2310.11441, 2023.
35. Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.[12045, 2024](https://md.research.elamur.ai/free/en/papers/yao-2024-tau-bench-benchmark-tool/).
36. Yoran, O., Amouyal, S. J., Malaviya, C., Bogin, B., Press, O., and Berant, J. [Assistantbench: Can web agents solve realistic and time-consuming tasks?](https://md.research.elamur.ai/free/en/papers/yoran-2024-assistantbench-can-web-agents-solve/), 2024. URL https://arxiv.org/abs/2407.15711. [↗](https://arxiv.org/abs/2407.15711)
37. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in Neural Information Processing Systems, 36:46595–46623, 2023.
38. Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al. [Webarena: A realistic web environment for building autonomous agents](https://md.research.elamur.ai/free/en/papers/zhou-2023-webarena-realistic-web-environment-building/). In The Twelfth International Conference on Learning Representations, 2023.
39. Zhou, X., Zhu, H., Mathur, L., Zhang, R., Yu, H., Qi, Z., Morency, L.-P., Bisk, Y., Fried, D., Neubig, G., and Sap, M. [SOTOPIA: Interactive evaluation for social intelligence in language agents](https://md.research.elamur.ai/free/en/papers/zhou-2023-sotopia-interactive-evaluation-social-intelligence/). In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/for…](https://openreview.net/for…) [↗](https://openreview.net/for…)

*Updated 2 November 2026*