--- title: "GPT-4V(ision) is a Generalist Web Agent, if — Zheng 2024" description: "The paper asks whether large multimodal models can serve as generalist web agents, meaning systems that carry out natural-language instructions on…" authors: ["Boyuan Zheng", "Boyu Gou", "Jihyung Kil", "Huan Sun", "Yu Su"] year: 2024 venue: "arXiv:2401.01614" source: "https://arxiv.org/abs/2401.01614" doi: "10.48550/arxiv.2401.01614" arxiv: "2401.01614" licence: "arXiv — non-exclusive license to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/zheng-2024-gpt-4v-ision-generalist-web/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zheng-2024-gpt-4v-ision-generalist-web/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / GPT-4V(ision) is a Generalist Web Agent, if Grounded # GPT-4V(ision) is a Generalist Web Agent, if Grounded **Authors:** [Boyuan Zheng](https://md.research.elamur.ai/free/en/authors/boyuan-zheng/), Boyu Gou, Jihyung Kil, [Huan Sun](https://md.research.elamur.ai/free/en/authors/huan-sun/), [Yu Su](https://md.research.elamur.ai/free/en/authors/yu-su/) · **Year:** 2024 · **Venue:** arXiv:2401.01614 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2401.01614) · [PDF](https://arxiv.org/pdf/2401.01614) · [DOI 10.48550/arxiv.2401.01614](https://doi.org/10.48550/arxiv.2401.01614) · [arXiv 2401.01614](https://arxiv.org/abs/2401.01614) · [OpenAlex](https://openalex.org/W4390601403) **Citations:** 9 citations (openalex-snapshot · 2026-10-14) · [121](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary The paper asks whether large multimodal models can serve as generalist web agents, meaning systems that carry out natural-language instructions on arbitrary real websites. The authors argue that rendered screenshots carry information more compactly than raw HTML and include content, such as embedded images, that HTML misses. They build SeeAct, which works in two stages. First, GPT-4V looks at a screenshot and writes a textual description of the next step (action generation). Second, that description is mapped to a concrete HTML element and operation (element grounding). The authors compare three grounding schemes: predicting element attributes, choosing among textual HTML candidates, and set-of-mark-style image labels. SeeAct is evaluated on Mind2Web in the usual offline setting with cached pages. The authors also built a new tool for running agents on live websites. When grounding is done by hand, GPT-4V completes about half of the live tasks, far ahead of text-only and fine-tuned baselines. Automatic grounding, however, still trails this oracle by a wide margin. The work matters because it identifies grounding, not planning, as the main obstacle for vision-based web agents. ## Key points - With oracle grounding, SeeAct on GPT-4V completes 51.1% of live-website tasks, compared with 13.3% for GPT-4 and 8.9% for FLAN-T5. Other baselines include BLIP-2, LLaVA-1.5 and CogAgent. - In one example page, 423 HTML elements take 186,490 GPT-2 tokens, while the same screenshot needs only 1,445 GPT-4V visual tokens. - The best grounding method combines HTML text with visuals. It outperforms image annotation (set-of-mark prompting) by up to roughly 10%, a method that caused heavy hallucination on webpages. Even so, it remains about 20–30% below oracle grounding. - In-context learning with large models generalizes better to unseen websites. Supervised fine-tuning still wins on websites seen during training. - Online and offline results diverge noticeably because a task often admits several valid plans. The authors therefore treat online evaluation as the more faithful measure. - Code, data and evaluation tools are released on GitHub (OSU-NLP-Group/SeeAct) under an OPEN-RAIL license. During live runs, every action was manually checked for safety before execution. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 121 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 115 not yet. - [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/chen-2024-image-worth-1-2-tokens/) - [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://md.research.elamur.ai/free/en/papers/he-2024-webvoyager-building-end-end-web/) - [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://md.research.elamur.ai/free/en/papers/qin-2025-ui-tars-pioneering-automated/) - [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://md.research.elamur.ai/free/en/papers/tseng-2024-two-tales-persona-llms-survey/) - [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://md.research.elamur.ai/free/en/papers/wu-2024-deepseek-vl2-mixture-experts/) - [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://md.research.elamur.ai/free/en/papers/xie-2024-osworld-benchmarking-multimodal-agents-open/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{zheng2024gpt, title = {{GPT-4V(ision) is a Generalist Web Agent, if Grounded}}, author = {Zheng, Boyuan and Gou, Boyu and Kil, Jihyung and Sun, Huan and Su, Yu}, year = {2024}, doi = {10.48550/arxiv.2401.01614}, eprint = {2401.01614}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2401.01614} } ``` - [← previous GPT-4o System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/) - [next → GPT系列大语言模型在自然语言处理任务中的鲁棒性](https://md.research.elamur.ai/free/en/papers/chen-2024-robustness-gpt-large/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) ## References 51 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Akter, S. N., Yu, Z., Muhamed, A., Ou, T., Bauerle, A., Cabrera, Á. A., Dholakia, K., Xiong, C., and Neubig, G. An in-depth look at gemini’s language abilities. 2023. URL [https://api.semanticscholar.org/CorpusID:266359502](https://api.semanticscholar.org/CorpusID:266359502). [↗](https://api.semanticscholar.org/CorpusID:266359502) 2. Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira,… [find](https://scholar.google.com/scholar?q=Alayrac%2C+J%2E%2DB%2E%2C+Donahue%2C+J%2E%2C+Luc%2C+P%2E%2C+Miech%2C+A%2E%2C+Barr%2C+I%2E%2C+Hasson%2C+Y%2E%2C+Lenc%2C+K%2E%2C+Mensch%2C+A%2E%2C+Millican%2C+K%2E%2C+Reynolds%2C+M%2E%2C+Ring%2C+R%2E%2C+Rutherford%2C+E%2E%2C+Cabi%2C+S%2E%2C+Han%2C+T%2E%2C+Gong%2C+Z%2E%2C+Samangooei%2C+S%2E%2C+Monteiro) [our card](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/) 3. Anil, G. T. G. R., Borgeaud, S., and et al., Y. W. Gemini: A family of highly capable multimodal models. 2023. URL [https://api.semanticscholar.org/CorpusID:266361876](https://api.semanticscholar.org/CorpusID:266361876). [↗](https://api.semanticscholar.org/CorpusID:266361876) [our card](https://md.research.elamur.ai/free/en/papers/gemini-2023-team-gemini-family-highly-capable-multimodal/) 4. Chandu, K. R., Bisk, Y., and Black, A. W. Grounding ‘grounding’in nlp. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4283– 4305, 2021. [find](https://scholar.google.com/scholar?q=Chandu%2C+K%2E+R%2E%2C+Bisk%2C+Y%2E%2C+and+Black%2C+A%2E+W%2E+Grounding+%E2%80%98grounding%E2%80%99in+nlp%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL%2DIJCNLP+2021%2C+pp%2E+4283%E2%80%93+4305%2C+2021%2E) 5. Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., and Zhao, R. Shikra: Unleashing multimodal llm’s referential dialogue magic. ArXiv, abs/2306.15195, 2023. URL [https://api.semanticscholar.org/CorpusID:259262082](https://api.semanticscholar.org/CorpusID:259262082). [↗](https://api.semanticscholar.org/CorpusID:259262082) [our card](https://md.research.elamur.ai/free/en/papers/chen-2023-shikra-unleashing-multimodal-llm-s/) 6. Cheng, K., Sun, Q., Chu, Y., Xu, F., Li, Y., Zhang, J., and Wu, Z. Seeclick: Harnessing gui grounding for advanced visual gui agents. 2024. URL [https://api.semanticscholar.org/CorpusID:267069082](https://api.semanticscholar.org/CorpusID:267069082). [↗](https://api.semanticscholar.org/CorpusID:267069082) [our card](https://md.research.elamur.ai/free/en/papers/cheng-2024-seeclick-harnessing-gui-grounding-advanced/) 7. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Valter, D., Narang, S., Mishra, G., Yu, A. W., Zhao, V., Huang, Y., Dai, A. M., Yu, H., Petrov, S., hsin Chi, E. H., Dean… [find](https://scholar.google.com/scholar?q=Chung%2C+H%2E+W%2E%2C+Hou%2C+L%2E%2C+Longpre%2C+S%2E%2C+Zoph%2C+B%2E%2C+Tay%2C+Y%2E%2C+Fedus%2C+W%2E%2C+Li%2C+E%2E%2C+Wang%2C+X%2E%2C+Dehghani%2C+M%2E%2C+Brahma%2C+S%2E%2C+Webson%2C+A%2E%2C+Gu%2C+S%2E+S%2E%2C+Dai%2C+Z%2E%2C+Suzgun%2C+M%2E%2C+Chen%2C+X%2E%2C+Chowdhery%2C+A%2E%2C+Valter%2C+D%2E%2C+Narang%2C+S) 8. Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. Mind2web: Towards a generalist agent for the web. arXiv preprint arXiv:2306.06070, 2023. [find](https://scholar.google.com/scholar?q=Deng%2C+X%2E%2C+Gu%2C+Y%2E%2C+Zheng%2C+B%2E%2C+Chen%2C+S%2E%2C+Stevens%2C+S%2E%2C+Wang%2C+B%2E%2C+Sun%2C+H%2E%2C+and+Su%2C+Y%2E+Mind2web%3A+Towards+a+generalist+agent+for+the+web%2E+arXiv+preprint+arXiv%3A2306%2E06070%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/deng-2023-mind2web-towards-generalist-agent-web/) 9. Furuta, H., Nachum, O., Lee, K.-H., Matsuo, Y., Gu, S. S., and Gur, I. Multimodal web navigation with instruction-finetuned foundation models. ArXiv, abs/2305.11854, 2023. URL [https://api.semanticscholar.org/CorpusID:258823350](https://api.semanticscholar.org/CorpusID:258823350). — [arXiv:2305.11854](https://arxiv.org/abs/2305.11854) [↗](https://api.semanticscholar.org/CorpusID:258823350) 10. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. International Journal of Computer Vision, 127:398 – 414, 2016. URL [https://api.semanticscholar.org/CorpusID:8081284](https://api.semanticscholar.org/CorpusID:8081284). [↗](https://api.semanticscholar.org/CorpusID:8081284) [our card](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/) 11. Gu, Y., Deng, X., and Su, Y. Don’t generate, discriminate: A proposal for grounding language models to real-world environments. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.… [find](https://scholar.google.com/scholar?q=Gu%2C+Y%2E%2C+Deng%2C+X%2E%2C+and+Su%2C+Y%2E+Don%E2%80%99t+generate%2C+discriminate%3A+A+proposal+for+grounding+language+models+to+real%2Dworld+environments%2E+In+Rogers%2C+A%2E%2C+Boyd%2DGraber%2C+J%2E%2C+and+Okazaki%2C+N%2E+%28eds%2E%29%2C+Proceedings+of+t) 12. Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., Manocha, D., and Zhou, T. Hallusionbench: An advanced diagnostic suite for entangled language hallucination&visual illusion in large vision-language models. 2023. URL [https://api.semanticscholar](https://api.semanticscholar). org/Cor… [↗](https://api.semanticscholar) 13. Gur, I., Nachum, O., Miao, Y., Safdari, M., Huang, A., Chowdhery, A., Narang, S., Fiedel, N., and Faust, A. Understanding html with large language models. In Conference on Empirical Methods in Natural Language Processing, 2022. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 252780086. [↗](https://api.semanticscholar.org/CorpusID:) 14. Gur, I., Furuta, H., Huang, A., Safdari, M., Matsuo, Y., Eck, D., and Faust, A. A real-world webagent with planning, long context understanding, and program synthesis. ArXiv, abs/2307.12856, 2023. URL [https://api.semanticscholar.org/CorpusID:260126067](https://api.semanticscholar.org/CorpusID:260126067). [↗](https://api.semanticscholar.org/CorpusID:260126067) 15. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D. X., and Steinhardt, J. Measuring massive multi-task language understanding. ArXiv, abs/2009.03300, 2020. URL [https://api.semanticscholar.org/CorpusID:221516475](https://api.semanticscholar.org/CorpusID:221516475). [↗](https://api.semanticscholar.org/CorpusID:221516475) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/) 16. Hong, W., Wang, W., Lv, Q., Xu, J., Yu, W., Ji, J., Wang, Y., Wang, Z., Dong, Y., Ding, M., and Tang, J. Cogagent: A visual language model for gui agents. 2023. URL [https://api.semanticscholar.org/CorpusID:266210390](https://api.semanticscholar.org/CorpusID:266210390). [↗](https://api.semanticscholar.org/CorpusID:266210390) [our card](https://md.research.elamur.ai/free/en/papers/hong-2023-cogagent-visual-language-model/) 17. Kazemzadeh, S., Ordonez, V., andre Matten, M., and Berg, T. L. Referitgame: Referring to objects in photographs of natural scenes. In Conference on Empirical Methods in Natural Language Processing, 2014. URL [https://api.semanticscholar.org/CorpusID:6308361](https://api.semanticscholar.org/CorpusID:6308361). [↗](https://api.semanticscholar.org/CorpusID:6308361) [our card](https://md.research.elamur.ai/free/en/papers/kazemzadeh-2014-referitgame-referring-objects-photographs-natural/) 18. Kim, G., Baldi, P., and McAleer, S. M. Language models can solve computer tasks. ArXiv, abs/2303.17491, 2023. URL [https://api.semanticscholar.org/CorpusID:257834038](https://api.semanticscholar.org/CorpusID:257834038). [↗](https://api.semanticscholar.org/CorpusID:257834038) [our card](https://md.research.elamur.ai/free/en/papers/kim-2023-language-models-solve-computer-tasks/) 19. Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M. C., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., and Fried, D. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. 2024. URL [https://api.semanticscholar.org/CorpusID:267199749](https://api.semanticscholar.org/CorpusID:267199749). [↗](https://api.semanticscholar.org/CorpusID:267199749) [our card](https://md.research.elamur.ai/free/en/papers/koh-2024-visualwebarena-evaluating-multimodal-agents-realistic/) 20. Lee, K., Joshi, M., Turc, I., Hu, H., Liu, F., Eisenschlos, J. M., Khandelwal, U., Shaw, P., Chang, M.-W., and Toutanova, K. Pix2struct: Screenshot parsing as pretraining for visual language understanding. ArXiv, abs/2210.03347, 2022. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 252762394. [↗](https://api.semanticscholar.org/CorpusID:) 21. Li, J., Li, D., Savarese, S., and Hoi, S. C. H. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ArXiv, abs/2301.12597, 2023. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 256390509. [↗](https://api.semanticscholar.org/CorpusID:) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/) 22. Lin, T.-Y., Maire, M., Belongie, S. J., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014. URL [https://api.semanticscholar.org/CorpusID:14113767](https://api.semanticscholar.org/CorpusID:14113767). [↗](https://api.semanticscholar.org/CorpusID:14113767) [our card](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/) 23. Liu, E. Z., Guu, K., Pasupat, P., Shi, T., and Liang, P. Reinforcement learning on web interfaces using workflow-guided exploration. In International Conference on Learning Representations (ICLR), 2018. URL https://arxiv.org/abs/1802.08802. [↗](https://arxiv.org/abs/1802.08802) 24. Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning, 2023a. [find](https://scholar.google.com/scholar?q=Liu%2C+H%2E%2C+Li%2C+C%2E%2C+Li%2C+Y%2E%2C+and+Lee%2C+Y%2E+J%2E+Improved+baselines+with+visual+instruction+tuning%2C+2023a%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/) 25. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. ArXiv, abs/2304.08485, 2023b. URL [https://api.semanticscholar.org/CorpusID:258179774](https://api.semanticscholar.org/CorpusID:258179774). [↗](https://api.semanticscholar.org/CorpusID:258179774) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/) 26. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning, 2023c. [find](https://scholar.google.com/scholar?q=Liu%2C+H%2E%2C+Li%2C+C%2E%2C+Wu%2C+Q%2E%2C+and+Lee%2C+Y%2E+J%2E+Visual+instruction+tuning%2C+2023c%2E) 27. Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Gu, Y., Ding, H., Men, K., Yang, K., Zhang, S., Deng, X., Zeng, A., Du, Z., Zhang, C., Shen, S., Zhang, T., Su, Y., Sun, H., Huang, M., Dong, Y., and Tang, J. Agentbench: Evaluating llms as agents. ArXiv, abs/2308.03688, 2023d. URL https:… [find](https://scholar.google.com/scholar?q=Liu%2C+X%2E%2C+Yu%2C+H%2E%2C+Zhang%2C+H%2E%2C+Xu%2C+Y%2E%2C+Lei%2C+X%2E%2C+Lai%2C+H%2E%2C+Gu%2C+Y%2E%2C+Gu%2C+Y%2E%2C+Ding%2C+H%2E%2C+Men%2C+K%2E%2C+Yang%2C+K%2E%2C+Zhang%2C+S%2E%2C+Deng%2C+X%2E%2C+Zeng%2C+A%2E%2C+Du%2C+Z%2E%2C+Zhang%2C+C%2E%2C+Shen%2C+S%2E%2C+Zhang%2C+T%2E%2C+Su%2C+Y%2E%2C+Sun%2C+H%2E%2C+Huang%2C+M%2E%2C+Do) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-agentbench-evaluating-llms-agents/) 28. Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. ArXiv, abs/2209.09513, 2022. URL [https://api.semanticscholar.org/CorpusID:252383606](https://api.semanticscholar.org/CorpusID:252383606). — [arXiv:2209.09513](https://arxiv.org/abs/2209.09513) [↗](https://api.semanticscholar.org/CorpusID:252383606) 29. Mazumder, S. and Riva, O. Flin: A flexible natural language interface for web navigation. ArXiv, abs/2010.12844, 2020. URL [https://api.semanticscholar.org/CorpusID:225067907](https://api.semanticscholar.org/CorpusID:225067907). [↗](https://api.semanticscholar.org/CorpusID:225067907) 30. OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. URL [https://api.semanticscholar.org/CorpusID:257532815](https://api.semanticscholar.org/CorpusID:257532815). [↗](https://api.semanticscholar.org/CorpusID:257532815) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/) 31. Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., and Wei, F. Kosmos-2: Grounding multimodal large language models to the world. ArXiv, abs/2306.14824, 2023. URL [https://api.semanticscholar.org/CorpusID:259262263](https://api.semanticscholar.org/CorpusID:259262263). [↗](https://api.semanticscholar.org/CorpusID:259262263) [our card](https://md.research.elamur.ai/free/en/papers/peng-2023-kosmos-2-grounding-multimodal-large/) 32. Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., and Lazebnik, S. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. International Journal of Computer Vision, 123:74 – 93, 2015. URL https://api. semanticscholar.org/CorpusI… [find](https://scholar.google.com/scholar?q=Plummer%2C+B%2E+A%2E%2C+Wang%2C+L%2E%2C+Cervantes%2C+C%2E+M%2E%2C+Caicedo%2C+J%2E+C%2E%2C+Hockenmaier%2C+J%2E%2C+and+Lazebnik%2C+S%2E+Flickr30k+entities%3A+Collecting+region%2Dto%2Dphrase+correspondences+for+richer+image%2Dto%2Dsentence+models%2E+Inter) 33. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PMLR, 2021. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Kim%2C+J%2E+W%2E%2C+Hallacy%2C+C%2E%2C+Ramesh%2C+A%2E%2C+Goh%2C+G%2E%2C+Agarwal%2C+S%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+Mishkin%2C+P%2E%2C+Clark%2C+J%2E%2C+et+al%2E+Learning+transferable+visual+models+from+natural+language+supervision%2E+In) [our card](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/) 34. Saikh, T., Ghosal, T., Mittal, A., Ekbal, A., and Bhattacharyya, P. Scienceqa: a novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23:289 – 301, 2022. URL [https://api.semanticscholar.org/CorpusID:250729995](https://api.semanticscholar.org/CorpusID:250729995). [↗](https://api.semanticscholar.org/CorpusID:250729995) 35. Shaw, P., Joshi, M., Cohan, J., Berant, J., Pasupat, P., Hu, H., Khandelwal, U., Lee, K., and Toutanova, K. From pixels to ui actions: Learning to follow instructions via graphical user interfaces. ArXiv, abs/2306.00245, 2023. URL [https://api.semanticscholar.org/CorpusID:258999511](https://api.semanticscholar.org/CorpusID:258999511). — [arXiv:2306.00245](https://arxiv.org/abs/2306.00245) [↗](https://api.semanticscholar.org/CorpusID:258999511) 36. Shen, S., Li, L. H., Tan, H., Bansal, M., Rohrbach, A., Chang, K.-W., Yao, Z., and Keutzer, K. How much can clip benefit vision-and-language tasks? arXiv preprint arXiv:2107.06383, 2021. — [arXiv:2107.06383](https://arxiv.org/abs/2107.06383) [find](https://scholar.google.com/scholar?q=Shen%2C+S%2E%2C+Li%2C+L%2E+H%2E%2C+Tan%2C+H%2E%2C+Bansal%2C+M%2E%2C+Rohrbach%2C+A%2E%2C+Chang%2C+K%2E%2DW%2E%2C+Yao%2C+Z%2E%2C+and+Keutzer%2C+K%2E+How+much+can+clip+benefit+vision%2Dand%2Dlanguage+tasks%3F+arXiv+preprint+arXiv%3A2107%2E06383%2C+2021%2E+%E2%80%94+%5BarXiv%3A2107) 37. Shi, T., Karpathy, A., Fan, L. J., Hernández, J. Z., and Liang, P. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, 2017. URL [https://api.semanticscholar.org/CorpusID:34953552](https://api.semanticscholar.org/CorpusID:34953552). [↗](https://api.semanticscholar.org/CorpusID:34953552) 38. Shtedritski, A., Rupprecht, C., and Vedaldi, A. What does clip know about a red circle? visual prompt engineering for vlms. ArXiv, abs/2304.06712, 2023. URL [https://api.semanticscholar.org/CorpusID:258108138](https://api.semanticscholar.org/CorpusID:258108138). — [arXiv:2304.06712](https://arxiv.org/abs/2304.06712) [↗](https://api.semanticscholar.org/CorpusID:258108138) 39. Sridhar, A., Lo, R., Xu, F. F., Zhu, H., and Zhou, S. Hierarchical prompting assists large language model on web navigation. ArXiv, abs/2305.14257, 2023. URL [https://api.semanticscholar.org/CorpusID:258841249](https://api.semanticscholar.org/CorpusID:258841249). [↗](https://api.semanticscholar.org/CorpusID:258841249) 40. Yan, A., Yang, Z., Zhu, W., Lin, K., Li, L., Wang, J., Yang, J., Zhong, Y., McAuley, J., Gao, J., Liu, Z., and Wang, L. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation. 2023. URL [https://api.semanticscholar.org/CorpusID:265149992](https://api.semanticscholar.org/CorpusID:265149992). [↗](https://api.semanticscholar.org/CorpusID:265149992) 41. Yang, J., Zhang, H., Li, F., Zou, X., yue Li, C., and Gao, J. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. ArXiv, abs/2310.11441, 2023a. URL [https://api.semanticscholar.org/CorpusID:264172201](https://api.semanticscholar.org/CorpusID:264172201). [↗](https://api.semanticscholar.org/CorpusID:264172201) 42. Yang, L., Wang, Y., Li, X., Wang, X., and Yang, J. Fine-grained visual prompting. ArXiv, abs/2306.04356, 2023b. URL [https://api.semanticscholar.org/CorpusID:259096008](https://api.semanticscholar.org/CorpusID:259096008). — [arXiv:2306.04356](https://arxiv.org/abs/2306.04356) [↗](https://api.semanticscholar.org/CorpusID:259096008) 43. Yang, Z., Li, L., Lin, K., Wang, J., Lin, C.-C., Liu, Z., and Wang, L. The dawn of lmms: Preliminary explorations with gpt-4v(ision). ArXiv, abs/2309.17421, 2023c. URL [https://api.semanticscholar.org/CorpusID:263310951](https://api.semanticscholar.org/CorpusID:263310951). [↗](https://api.semanticscholar.org/CorpusID:263310951) [our card](https://md.research.elamur.ai/free/en/papers/yang-2023-dawn-lmms-preliminary-explorations-gpt/) 44. Yao, S., Chen, H., Yang, J., and Narasimhan, K. Webshop: Towards scalable real-world web interaction with grounded language agents. ArXiv, abs/2207.01206, 2022. URL [https://api.semanticscholar.org/CorpusID:250264533](https://api.semanticscholar.org/CorpusID:250264533). [↗](https://api.semanticscholar.org/CorpusID:250264533) [our card](https://md.research.elamur.ai/free/en/papers/yao-2022-webshop-towards-scalable-real-world/) 45. You, H., Zhang, H., Gan, Z., Du, X., Zhang, B., Wang, Z., Cao, L., Chang, S.-F., and Yang, Y. Ferret: Refer and ground anything anywhere at any granularity. ArXiv, abs/2310.07704, 2023. URL [https://api.semanticscholar.org/CorpusID:263834718](https://api.semanticscholar.org/CorpusID:263834718). — [arXiv:2310.07704](https://arxiv.org/abs/2310.07704) [↗](https://api.semanticscholar.org/CorpusID:263834718) 46. Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark f… [find](https://scholar.google.com/scholar?q=Yue%2C+X%2E%2C+Ni%2C+Y%2E%2C+Zhang%2C+K%2E%2C+Zheng%2C+T%2E%2C+Liu%2C+R%2E%2C+Zhang%2C+G%2E%2C+Stevens%2C+S%2E%2C+Jiang%2C+D%2E%2C+Ren%2C+W%2E%2C+Sun%2C+Y%2E%2C+Wei%2C+C%2E%2C+Yu%2C+B%2E%2C+Yuan%2C+R%2E%2C+Sun%2C+R%2E%2C+Yin%2C+M%2E%2C+Zheng%2C+B%2E%2C+Yang%2C+Z%2E%2C+Liu%2C+Y%2E%2C+Huang%2C+W%2E%2C+Sun%2C+H%2E%2C+Su%2C+) [our card](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/) 47. Zhang, X., Lu, Y., Wang, W., Yan, A., Yan, J., Qin, L., Wang, H., Yan, X., Wang, W. Y., and Petzold, L. R. Gpt-4v(ision) as a generalist evaluator for vision-language tasks. ArXiv, abs/2311.01361, 2023. URL [https://api.semanticscholar.org/CorpusID:264935635](https://api.semanticscholar.org/CorpusID:264935635). — [arXiv:2311.01361](https://arxiv.org/abs/2311.01361) [↗](https://api.semanticscholar.org/CorpusID:264935635) 48. Zhao, Y., Lin, Z., Zhou, D., Huang, Z., Feng, J., and Kang, B. Bubogpt: Enabling visual grounding in multi-modal llms. ArXiv, abs/2307.08581, 2023. URL [https://api.semanticscholar.org/CorpusID:259937702](https://api.semanticscholar.org/CorpusID:259937702). — [arXiv:2307.08581](https://arxiv.org/abs/2307.08581) [↗](https://api.semanticscholar.org/CorpusID:259937702) 49. Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A. S. S., Chen, W., and Duan, N. Agieval: A human-centric benchmark for evaluating foundation models. ArXiv, abs/2304.06364, 2023. URL [https://api.semanticscholar.org/CorpusID:258108259](https://api.semanticscholar.org/CorpusID:258108259). [↗](https://api.semanticscholar.org/CorpusID:258108259) [our card](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/) 50. Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A. Scene parsing through ade20k dataset. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5122–5130, 2017. URL [https://api.semanticscholar.org/CorpusID:5636055](https://api.semanticscholar.org/CorpusID:5636055). [↗](https://api.semanticscholar.org/CorpusID:5636055) 51. Table of Content: 1. Appendix A: Offline Experiments Method Details 2. Appendix B: Markup Type Ablation Study 3. Appendix C: Online Experiment Details 4. Appendix D:Offline Experiment Prompts 5. Appendix E: Error Examples for Grounding via Image Annotation 6. Appendix F: Strong Capability of Plannin… [find](https://scholar.google.com/scholar?q=Table+of+Content%3A+1%2E+Appendix+A%3A+Offline+Experiments+Method+Details+2%2E+Appendix+B%3A+Markup+Type+Ablation+Study+3%2E+Appendix+C%3A+Online+Experiment+Details+4%2E+Appendix+D%3AOffline+Experiment+Prompts+5%2E+Appen)