---
title: "AppAgent: Multimodal Agents as Smartphone Users (2023)"
description: "This Tencent paper presents AppAgent, a framework in which a vision-capable multimodal LLM… 35 references, cited by 96 papers here."
authors: ["Chi Zhang", "Zhao Yang", "Jiaxuan Liu", "Yanda Li", "Yucheng Han", "Xin Chen", "Zebiao Huang", "Bin Fu", "Gang Yu"]
year: 2023
venue: "arXiv:2312.13771"
source: "https://arxiv.org/abs/2312.13771"
doi: "10.48550/arxiv.2312.13771"
arxiv: "2312.13771"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2023-appagent-multimodal-agents-smartphone-users/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2023-appagent-multimodal-agents-smartphone-users/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / AppAgent: Multimodal Agents as Smartphone Users

# AppAgent: Multimodal Agents as Smartphone Users

**Authors:** [Chi Zhang](https://md.research.elamur.ai/free/en/authors/chi-zhang/), [Zhao Yang](https://md.research.elamur.ai/free/en/authors/zhao-yang/), Jiaxuan Liu, Yanda Li, [Yucheng Han](https://md.research.elamur.ai/free/en/authors/yucheng-han/), [Xin Chen](https://md.research.elamur.ai/free/en/authors/xin-chen/), Zebiao Huang, [Bin Fu](https://md.research.elamur.ai/free/en/authors/bin-fu/) and [Gang Yu](https://md.research.elamur.ai/free/en/authors/gang-yu/) · **Year:** 2023 · **Venue:** arXiv:2312.13771 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2312.13771) · [PDF](https://arxiv.org/pdf/2312.13771) · [DOI 10.48550/arxiv.2312.13771](https://doi.org/10.48550/arxiv.2312.13771) · [arXiv 2312.13771](https://arxiv.org/abs/2312.13771) · [OpenAlex](https://openalex.org/W4390136788)

**Citations:** 11 citations (OpenAlex, 2026-10-14) · [96](https://md.research.elamur.ai/free/en/papers/zhang-2023-appagent-multimodal-agents-smartphone-users/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most LLM-based agents work only with text, and phone assistants such as Siri depend on back-end system access and function calls. This Tencent paper presents AppAgent, a framework in which a vision-capable multimodal LLM operates smartphone apps entirely through the graphical interface, as a person would. Interactive screen elements receive numeric tags. The model therefore selects an element by its label and applies a simple action, such as a tap, text entry or swipe, instead of predicting raw screen coordinates.

Learning happens in two stages. In the exploration stage, the agent either experiments with an app on its own or watches a few human demonstrations, and it records the function of each UI element in an app-specific document. In the deployment stage, it consults that document to carry out user requests. The approach needs no fine-tuning and no large per-app training data.

The authors tested the agent on 50 tasks across 10 apps. Both the restricted action space and the generated documents clearly improve on a plain GPT-4 baseline. The work points to a low-cost, app-agnostic route to phone automation.

## Key points
- **Evaluation scope:** The benchmark covers 50 tasks in 10 apps, including Telegram, YouTube, TEMU, Clock, an email client, maps and Adobe Lightroom. The quantitative comparison (Table 1) uses 45 tasks from 9 apps. Lightroom was left out because task success there is hard to judge objectively.
- **Action space:** Actions are label-based functions such as `tap(element)`, `text("...")` and `swipe(element, direction, distance)`. The authors note that the LLM is poor at producing accurate xy coordinates, and this design removes that requirement, which substantially raises the GPT-4 baseline's performance.
- **Documents:** Documents built through autonomous exploration and documents built from observed demonstrations are both reported to improve performance over the baseline without documents.
- **Lightroom case study:** The agent edited five images with defects such as low contrast and overexposure. It received a generic instruction to fix each image, with no description of the problem. A user study ranked the results:
  - Agents with documents beat the GPT-4 baseline.
  - Documents derived from demonstrations performed about as well as hand-written ones.
  - Agents with documents used a wider range of editing tools.
- **Limitations and availability:** Multi-touch and irregular gestures are not supported. The framework is open-sourced (appagent-official.github.io).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🖱️ Human-Computer Interaction](https://md.research.elamur.ai/free/en/shelves/computer-science/human-computer-interaction/) · 160 works

🧭 Topics (1): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works

## Cite

Cite as: Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu and Gang Yu. 2023. AppAgent: Multimodal Agents as Smartphone Users. arXiv:2312.13771.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhang2023appagent,
  title = {{AppAgent: Multimodal Agents as Smartphone Users}},
  author = {Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu and Gang Yu},
  year = {2023},
  doi = {10.48550/arxiv.2312.13771},
  eprint = {2312.13771},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.13771}
}
```

- [← previous SemDeDup: Data-efficient learning at web-scale through semantic deduplication](https://md.research.elamur.ai/free/en/papers/abbas-2023-semdedup-data-efficient-learning-web/)
- [next → AutoAgents: A Framework for Automatic Agent Generation](https://md.research.elamur.ai/free/en/papers/chen-2023-autoagents-framework-automatic-agent-generation/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

35 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil… — [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://md.research.elamur.ai/free/en/papers/ahn-2022-i-i-say-grounding-language/)
2. Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. 2023. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818. [find](https://scholar.google.com/scholar?q=Anthony+Brohan%2C+Noah+Brown%2C+Justice+Carbajal%2C+Yevgen+Chebotar%2C+Xi+Chen%2C+Krzysztof+Choromanski%2C+Tianli+Ding%2C+Danny+Driess%2C+Avinava+Dubey%2C+Chelsea+Finn%2C+et+al%2E+2023%2E+Rt%2D2%3A+Vision%2Dlanguage%2Daction+models+)
3. Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. 2022. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817. [find](https://scholar.google.com/scholar?q=Anthony+Brohan%2C+Noah+Brown%2C+Justice+Carbajal%2C+Yevgen+Chebotar%2C+Joseph+Dabis%2C+Chelsea+Finn%2C+Keerthana+Gopalakrishnan%2C+Karol+Hausman%2C+Alex+Herzog%2C+Jasmine+Hsu%2C+et+al%2E+2022%2E+Rt%2D1%3A+Robotics+transformer+fo)
4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
5. Meta FAIR, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. 2022. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378(6624):1067–1074. [find](https://scholar.google.com/scholar?q=Meta+FAIR%2C+Anton+Bakhtin%2C+Noam+Brown%2C+Emily+Dinan%2C+Gabriele+Farina%2C+Colin+Flaherty%2C+Daniel+Fried%2C+Andrew+Goff%2C+Jonathan+Gray%2C+Hengyuan+Hu%2C+et+al%2E+2022%2E+Human%2Dlevel+play+in+the+game+of+diplomacy+by+com)
6. Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. 2023. Multimodal web navigation with instruction-finetuned foundation models. [find](https://scholar.google.com/scholar?q=Hiroki+Furuta%2C+Kuang%2DHuei+Lee%2C+Ofir+Nachum%2C+Yutaka+Matsuo%2C+Aleksandra+Faust%2C+Shixiang+Shane+Gu%2C+and+Izzeddin+Gur%2E+2023%2E+Multimodal+web+navigation+with+instruction%2Dfinetuned+foundation+models%2E)
7. Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. 2023. [A real-world webagent with planning, long context understanding, and program synthesis](https://md.research.elamur.ai/free/en/papers/gur-2023-real-world-webagent-planning-long/).
8. Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. 2023. [Chartllama: A multimodal llm for chart understanding and generation](https://md.research.elamur.ai/free/en/papers/han-2023-chartllama-multimodal-llm-chart-understanding/).
9. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2023. [Metagpt: Meta programming for a multi-agent collaborative framework](https://md.research.elamur.ai/free/en/papers/hong-2023-metagpt-meta-programming-multi/).
10. Zhiting Hu and Tianmin Shu. 2023. Language models, agent models, and world models: The law for machine reasoning and planning. arXiv preprint arXiv:2312.05230. [find](https://scholar.google.com/scholar?q=Zhiting+Hu+and+Tianmin+Shu%2E+2023%2E+Language+models%2C+agent+models%2C+and+world+models%3A+The+law+for+machine+reasoning+and+planning%2E+arXiv+preprint+arXiv%3A2312%2E05230%2E)
11. Yanda Li, Chi Zhang, Gang Yu, Zhibin Wang, Bin Fu, Guosheng Lin, Chunhua Shen, Ling Chen, and Yunchao Wei. 2023. Stablellava: Enhanced visual instruction tuning with synthesized image-dialogue data. [find](https://scholar.google.com/scholar?q=Yanda+Li%2C+Chi+Zhang%2C+Gang+Yu%2C+Zhibin+Wang%2C+Bin+Fu%2C+Guosheng+Lin%2C+Chunhua+Shen%2C+Ling+Chen%2C+and+Yunchao+Wei%2E+2023%2E+Stablellava%3A+Enhanced+visual+instruction+tuning+with+synthesized+image%2Ddialogue+data%2E)
12. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/).
13. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/).
14. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2023c. [Agent-Bench: Evaluating LLM…](https://md.research.elamur.ai/free/en/papers/liu-2023-agentbench-evaluating-llms-agents/)
15. OpenAI. 2021. Chatgpt. research/chatgpt. [https://openai.com/](https://openai.com/) [↗](https://openai.com/)
16. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
17. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–22. [find](https://scholar.google.com/scholar?q=Joon+Sung+Park%2C+Joseph+O%E2%80%99Brien%2C+Carrie+Jun+Cai%2C+Meredith+Ringel+Morris%2C+Percy+Liang%2C+and+Michael+S+Bernstein%2E+2023%2E+Generative+agents%3A+Interactive+simulacra+of+human+behavior%2E+In+Proceedings+of+the+36)
18. Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. 2023. [Communicative agents for software development](https://md.research.elamur.ai/free/en/papers/qian-2023-chatdev-communicative-agents-software/). arXiv preprint arXiv:2307.07924.
19. Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. 2022. A generalist agent. arXiv preprint arXiv:2205.06175. [find](https://scholar.google.com/scholar?q=Scott+Reed%2C+Konrad+Zolna%2C+Emilio+Parisotto%2C+Sergio+Gomez+Colmenarejo%2C+Alexander+Novikov%2C+Gabriel+Barth%2DMaron%2C+Mai+Gimenez%2C+Yury+Sulsky%2C+Jackie+Kay%2C+Jost+Tobias+Springenberg%2C+et+al%2E+2022%2E+A+generalist+)
20. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Yongliang+Shen%2C+Kaitao+Song%2C+Xu+Tan%2C+Dongsheng+Li%2C+Weiming+Lu%2C+and+Yueting+Zhuang%2E+2023%2E+Hugginggpt%3A+Solving+ai+tasks+with+chatgpt+and+its+friends+in+huggingface%2E+In+Advances+in+Neural+Information+Pro)
21. Chunyi Sun, Junlin Han, Weijian Deng, Xinlong Wang, Zishan Qin, and Stephen Gould. 2023. 3d-gpt: Procedural 3d modeling with large language models. arXiv preprint arXiv:2310.12945. — [arXiv:2310.12945](https://arxiv.org/abs/2310.12945) [find](https://scholar.google.com/scholar?q=Chunyi+Sun%2C+Junlin+Han%2C+Weijian+Deng%2C+Xinlong+Wang%2C+Zishan+Qin%2C+and+Stephen+Gould%2E+2023%2E+3d%2Dgpt%3A+Procedural+3d+modeling+with+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E12945%2E+%E2%80%94+%5BarXiv%3A2310%2E12945)
22. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca). [↗](https://github.com/tatsu-lab/stanford_alpaca)
23. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/).
24. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
25. Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, Xiaojian Ma, and Yitao Liang. 2023. Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. [find](https://scholar.google.com/scholar?q=Zihao+Wang%2C+Shaofei+Cai%2C+Anji+Liu%2C+Yonggang+Jin%2C+Jinbing+Hou%2C+Bowei+Zhang%2C+Haowei+Lin%2C+Zhaofeng+He%2C+Zilong+Zheng%2C+Yaodong+Yang%2C+Xiaojian+Ma%2C+and+Yitao+Liang%2E+2023%2E+Jarvis%2D1%3A+Open%2Dworld+multi%2Dtask+agen)
26. Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023. [The rise and potential of large language model based agents: A survey](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/). arXiv preprint arXiv:2309.07864.
27. Tianbao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng, Yitao Liu, Toh Jing Hua, Junning Zhao, Qian Liu, Che Liu, Leo Z. Liu, Yiheng Xu, Hongjin Su, Dongchan Shin, Caiming Xiong, and Tao Yu. 2023. [Openagents: An open platform for language agents in the wild](https://md.research.elamur.ai/free/en/papers/xie-2023-openagents-open-platform-language-agents/).
28. Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. 2023. [Exploring large language models for communication games: An empirical study on werewolf](https://md.research.elamur.ai/free/en/papers/xu-2023-exploring-large-language-models-communication/). arXiv preprint arXiv:2309.04658.
29. An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, Zicheng Liu, and Lijuan Wang. 2023. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation. arXiv preprint arXiv: 2311.07562. — [arXiv:2311.07562](https://arxiv.org/abs/2311.07562) [find](https://scholar.google.com/scholar?q=An+Yan%2C+Zhengyuan+Yang%2C+Wanrong+Zhu%2C+Kevin+Lin%2C+Linjie+Li%2C+Jianfeng+Wang%2C+Jianwei+Yang%2C+Yiwu+Zhong%2C+Julian+McAuley%2C+Jianfeng+Gao%2C+Zicheng+Liu%2C+and+Lijuan+Wang%2E+2023%2E+Gpt%2D4v+in+wonderland%3A+Large+multim)
30. Hui Yang, Sifu Yue, and Yunzhong He. 2023a. Autogpt for online decision making: Benchmarks and additional opinions. [find](https://scholar.google.com/scholar?q=Hui+Yang%2C+Sifu+Yue%2C+and+Yunzhong+He%2E+2023a%2E+Autogpt+for+online+decision+making%3A+Benchmarks+and+additional+opinions%2E)
31. Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023b. [The dawn of lmms: Preliminary explorations with gpt-4v (ision)](https://md.research.elamur.ai/free/en/papers/yang-2023-dawn-lmms-preliminary-explorations-gpt/). arXiv preprint arXiv:2309.17421.
32. Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023c. [The dawn of lmms: Preliminary explorations with gpt-4v(ision)](https://md.research.elamur.ai/free/en/papers/yang-2023-dawn-lmms-preliminary-explorations-gpt/). arXiv preprint arXiv: 2309.17421.
33. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. [ReAct: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). In ICLR.
34. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. [Glm-130b: An open bilingual pre-trained model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/). arXiv preprint arXiv:2210.02414.
35. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/).

*Updated 2 November 2026*