---
title: "AVA: Attentive VLM Agent for Mastering StarCraft II (2026)"
description: "The authors build AVACraft, a benchmark that offers RGB screen and minimap images, textual battlefield descriptions and… 32 references."
authors: ["Weiyu Ma", "Yuqian Fu", "Zecheng Zhang", "Bernard Ghanem", "Guohao Li"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.208/"
doi: "10.18653/v1/2026.findings-acl.208"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ma-2026-ava-attentive-vlm-agent-mastering/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ma-2026-ava-attentive-vlm-agent-mastering/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/23/) / AVA: Attentive VLM Agent for Mastering StarCraft II

# AVA: Attentive VLM Agent for Mastering StarCraft II

**Authors:** Weiyu Ma, Yuqian Fu, Zecheng Zhang, [Bernard Ghanem](https://md.research.elamur.ai/free/en/authors/bernard-ghanem/) and [Guohao Li](https://md.research.elamur.ai/free/en/authors/guohao-li/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.208/) · [PDF](https://aclanthology.org/2026.findings-acl.208.pdf) · [DOI 10.18653/v1/2026.findings-acl.208](https://doi.org/10.18653/v1/2026.findings-acl.208) · [OpenAlex](https://openalex.org/W4417148752)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Most StarCraft II testbeds, such as SMAC and SMACv2, give agents abstract feature vectors and support only reinforcement learning. Vision-language models therefore cannot be tested on fine-grained unit control. The authors build AVACraft, a benchmark that offers RGB screen and minimap images, textual battlefield descriptions and structured unit data in one POMDP setting with Fog of War. Training-based and zero-shot agents can thus be compared under identical conditions. The benchmark keeps full unit abilities and uses adaptive opponents: the built-in AI plus scripted behavior-tree policies that are drawn at random. Baselines cover six MARL algorithms with Swin-Transformer encoders, each trained for 5M steps, and several proprietary and open VLMs used without training. VLM agents reach 75–90% win rates zero-shot, while the best MARL result is only 19.3%. Blinded expert raters also judge VLM behavior as more coherent and human-like. The work maps out trade-offs in sample efficiency, interpretability and cost, and motivates hybrid RL–VLM agents.

## Key points
- AVACraft has 21 scenarios: 12 micromanagement, 5 coordination and 4 strategic. All support PvE and PvP, use sparse ±1/0 rewards and end at a 300-second time limit.
- MARL baselines are IQL, QMIX, QTRAN, VDN, MAPPO and IPPO. VLMs include GPT-4o, GPT-4-Turbo, GPT-4o-mini, Qwen3-VL-8B and Qwen3-VL-30B.
- The MARL agents did not benefit from added text input (IPPO Vision+Text did worse than vision alone). Every VLM scored 0% on the hardest maps.
- Seven raters, ranging from a professional player to a spectator, preferred VLM agents on all metrics (Mann-Whitney U, p \< 0.01).
- On cost, MARL training takes about 65–80 h on two A100s, with ~8–10 ms inference per step. GPT-4o needs ~2.3 s and ~$0.027 per decision, about $4.05 per episode. The Qwen3-VL models run on a single A100 with no API fees.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

## Cite

Cite as: Weiyu Ma, Yuqian Fu, Zecheng Zhang, Bernard Ghanem and Guohao Li. 2026. AVA: Attentive VLM Agent for Mastering StarCraft II. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ma2026ava,
  title = {{AVA: Attentive VLM Agent for Mastering StarCraft II}},
  author = {Weiyu Ma, Yuqian Fu, Zecheng Zhang, Bernard Ghanem and Guohao Li},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.208},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.208}
}
```

- [← previous AutoTaskEval: Towards Domain-Specific and Fine-Grained Evaluation for LLMs](https://md.research.elamur.ai/free/en/papers/lyu-2026-autotaskeval-domain-specific-fine-grained/)
- [next → AvarLab: An Integrated Digital Ecosystem for Avar, a Morphologically Rich Low-Resource Language](https://md.research.elamur.ai/free/en/papers/zagidov-2026-avarlab-integrated-digital-ecosystem-avar/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/23/)

## References

32 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. 2024. Claude 3 model card. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2024%2E+Claude+3+model+card%2E)
2. Jinze Bai et al. 2023. [Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/). arXiv preprint arXiv:2308.12966.
3. Anthony Brohan et al. 2023. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818. [find](https://scholar.google.com/scholar?q=Anthony+Brohan+et+al%2E+2023%2E+Rt%2D2%3A+Vision%2Dlanguage%2Daction+models+transfer+web+knowledge+to+robotic+control%2E+arXiv+preprint+arXiv%3A2307%2E15818%2E)
4. Shaofei Cai et al. 2023. Groot: Learning to follow instructions by watching gameplay videos. arXiv preprint arXiv:2310.08235. — [arXiv:2310.08235](https://arxiv.org/abs/2310.08235) [find](https://scholar.google.com/scholar?q=Shaofei+Cai+et+al%2E+2023%2E+Groot%3A+Learning+to+follow+instructions+by+watching+gameplay+videos%2E+arXiv+preprint+arXiv%3A2310%2E08235%2E+%E2%80%94+%5BarXiv%3A2310%2E08235%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E08235%29)
5. Yue Deng, Weiyu Ma, Yuxin Fan, Yin Zhang, Haifeng Zhang, and Jian Zhao. 2024a. A new approach to solving smac task: Generating decision tree code from large language models. Preprint, arXiv:2410.16024. — [arXiv:2410.16024](https://arxiv.org/abs/2410.16024) [find](https://scholar.google.com/scholar?q=Yue+Deng%2C+Weiyu+Ma%2C+Yuxin+Fan%2C+Yin+Zhang%2C+Haifeng+Zhang%2C+and+Jian+Zhao%2E+2024a%2E+A+new+approach+to+solving+smac+task%3A+Generating+decision+tree+code+from+large+language+models%2E+Preprint%2C+arXiv%3A2410%2E16024)
6. Yue Deng, Yan Yu, Weiyu Ma, Zirui Wang, Wenhui Zhu, Jian Zhao, and Yin Zhang. 2024b. Smac-hard: Enabling mixed opponent strategy script and self-play on smac. Preprint, arXiv:2412.17707. — [arXiv:2412.17707](https://arxiv.org/abs/2412.17707) [find](https://scholar.google.com/scholar?q=Yue+Deng%2C+Yan+Yu%2C+Weiyu+Ma%2C+Zirui+Wang%2C+Wenhui+Zhu%2C+Jian+Zhao%2C+and+Yin+Zhang%2E+2024b%2E+Smac%2Dhard%3A+Enabling+mixed+opponent+strategy+script+and+self%2Dplay+on+smac%2E+Preprint%2C+arXiv%3A2412%2E17707%2E+%E2%80%94+%5BarXiv%3A2412)
7. Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob N. Foerster, and Shimon Whiteson. 2023. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. Preprint, arXiv:2212.07489. — [arXiv:2212.07489](https://arxiv.org/abs/2212.07489) [find](https://scholar.google.com/scholar?q=Benjamin+Ellis%2C+Jonathan+Cook%2C+Skander+Moalla%2C+Mikayel+Samvelyan%2C+Mingfei+Sun%2C+Anuj+Mahajan%2C+Jakob+N%2E+Foerster%2C+and+Shimon+Whiteson%2E+2023%2E+Smacv2%3A+An+improved+benchmark+for+cooperative+multi%2Dagent+rei)
8. Lei Han, Jiechao Xiong, Peng Sun, Xinghai Sun, Meng Fang, Qingwei Guo, Qiaobo Chen, Tengfei Shi, Hongsheng Yu, Xipeng Wu, et al. 2020. Tstarbot-x: An open-sourced and comprehensive study for efficient league training in starcraft ii full game. arXiv preprint arXiv:2011.13729. — [arXiv:2011.13729](https://arxiv.org/abs/2011.13729) [find](https://scholar.google.com/scholar?q=Lei+Han%2C+Jiechao+Xiong%2C+Peng+Sun%2C+Xinghai+Sun%2C+Meng+Fang%2C+Qingwei+Guo%2C+Qiaobo+Chen%2C+Tengfei+Shi%2C+Hongsheng+Yu%2C+Xipeng+Wu%2C+et+al%2E+2020%2E+Tstarbot%2Dx%3A+An+open%2Dsourced+and+comprehensive+study+for+efficient)
9. Hongliang He et al. 2024. [Webvoyager: Building an end-to-end web agent with large multimodal models](https://md.research.elamur.ai/free/en/papers/he-2024-webvoyager-building-end-end-web/). arXiv preprint arXiv:2401.13919.
10. Ruozi Huang, Xipeng Wu, Hongsheng Yu, Zhong Fan, Haobo Fu, QIANG FU, and Yang Wei. 2023. A robust and opponent-aware league training method for starcraft ii. In Thirty-seventh Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Ruozi+Huang%2C+Xipeng+Wu%2C+Hongsheng+Yu%2C+Zhong+Fan%2C+Haobo+Fu%2C+QIANG+FU%2C+and+Yang+Wei%2E+2023%2E+A+robust+and+opponent%2Daware+league+training+method+for+starcraft+ii%2E+In+Thirty%2Dseventh+Conference+on+Neural+Inf)
11. Zongyuan Li, Yanan Ni, Runnan Qi, Lumin Jiang, Chang Lu, Xiaojie Xu, Xiangbei Liu, Pengfei Li, Yunzheng Guo, Zhe Ma, et al. 2024. Llm-pysc2: Starcraft ii learning environment for large language models. arXiv preprint arXiv:2411.05348. — [arXiv:2411.05348](https://arxiv.org/abs/2411.05348) [find](https://scholar.google.com/scholar?q=Zongyuan+Li%2C+Yanan+Ni%2C+Runnan+Qi%2C+Lumin+Jiang%2C+Chang+Lu%2C+Xiaojie+Xu%2C+Xiangbei+Liu%2C+Pengfei+Li%2C+Yunzheng+Guo%2C+Zhe+Ma%2C+et+al%2E+2024%2E+Llm%2Dpysc2%3A+Starcraft+ii+learning+environment+for+large+language+models)
12. Michael Mathieu, Sherjil Ozair, Srivatsan Srinivasan, Caglar Gulcehre, Shangtong Zhang, Ray Jiang, Tom Le Paine, Konrad Zolna, Richard Powell, Julian Schrittwieser, et al. 2021. Starcraft ii unplugged: Large scale offline reinforcement learning. In Deep RL Workshop NeurIPS 2021. [find](https://scholar.google.com/scholar?q=Michael+Mathieu%2C+Sherjil+Ozair%2C+Srivatsan+Srinivasan%2C+Caglar+Gulcehre%2C+Shangtong+Zhang%2C+Ray+Jiang%2C+Tom+Le+Paine%2C+Konrad+Zolna%2C+Richard+Powell%2C+Julian+Schrittwieser%2C+et+al%2E+2021%2E+Starcraft+ii+unplugged)
13. OpenAI. 2023a. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
14. OpenAI. 2023b. Gpt-4v(ision) system card. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023b%2E+Gpt%2D4v%28ision%29+system+card%2E)
15. Davide Paglieri et al. 2025. Balrog: Benchmarking agentic llm and vlm reasoning on games. ICLR. [find](https://scholar.google.com/scholar?q=Davide+Paglieri+et+al%2E+2025%2E+Balrog%3A+Benchmarking+agentic+llm+and+vlm+reasoning+on+games%2E+ICLR%2E)
16. Alec Radford et al. 2021. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In ICML.
17. Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. 2019. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043. — [arXiv:1902.04043](https://arxiv.org/abs/1902.04043) [find](https://scholar.google.com/scholar?q=Mikayel+Samvelyan%2C+Tabish+Rashid%2C+Christian+Schroeder+De+Witt%2C+Gregory+Farquhar%2C+Nantas+Nardelli%2C+Tim+GJ+Rudner%2C+Chia%2DMan+Hung%2C+Philip+HS+Torr%2C+Jakob+Foerster%2C+and+Shimon+Whiteson%2E+2019%2E+The+starcraft)
18. DI star Contributors. 2021. Di-star: An open-sourse reinforcement learning framework for starcraftii. [https://github.com/opendilab/DI-star](https://github.com/opendilab/DI-star). [↗](https://github.com/opendilab/DI-star)
19. Weihao Tan et al. 2024. Cradle: Empowering foundation agents towards general computer control. arXiv preprint arXiv:2403.03186. — [arXiv:2403.03186](https://arxiv.org/abs/2403.03186) [find](https://scholar.google.com/scholar?q=Weihao+Tan+et+al%2E+2024%2E+Cradle%3A+Empowering+foundation+agents+towards+general+computer+control%2E+arXiv+preprint+arXiv%3A2403%2E03186%2E+%E2%80%94+%5BarXiv%3A2403%2E03186%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2403%2E03186%29)
20. Gemini Team et al. 2023. [Gemini: A family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805.
21. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
22. Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. 2019. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350–354. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Igor+Babuschkin%2C+Wojciech+M+Czarnecki%2C+Micha%C3%ABl+Mathieu%2C+Andrew+Dudzik%2C+Junyoung+Chung%2C+David+H+Choi%2C+Richard+Powell%2C+Timo+Ewalds%2C+Petko+Georgiev%2C+et+al%2E+2019%2E+Grandmaster+level+in+starc)
23. Zihao Wang et al. 2025. Jarvis-vla: Post-training large-scale vision language models to play visual games with keyboards and mouse. arXiv preprint arXiv:2503.16365. [find](https://scholar.google.com/scholar?q=Zihao+Wang+et+al%2E+2025%2E+Jarvis%2Dvla%3A+Post%2Dtraining+large%2Dscale+vision+language+models+to+play+visual+games+with+keyboards+and+mouse%2E+arXiv+preprint+arXiv%3A2503%2E16365%2E)
24. Haotian Liu et al. 2023. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). NeurIPS.
25. Xinrun Xu et al. 2024. Mcu: An evaluation framework for open-ended game agents. arXiv preprint arXiv:2310.08367. — [arXiv:2310.08367](https://arxiv.org/abs/2310.08367) [find](https://scholar.google.com/scholar?q=Xinrun+Xu+et+al%2E+2024%2E+Mcu%3A+An+evaluation+framework+for+open%2Dended+game+agents%2E+arXiv+preprint+arXiv%3A2310%2E08367%2E+%E2%80%94+%5BarXiv%3A2310%2E08367%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E08367%29)
26. Weiyu Ma, Qirui Mi, Yongcheng Zeng, Xue Yan, Yuqiao Wu, Runji Lin, Haifeng Zhang, and Jun Wang. 2024. Large language models play starcraft ii: Benchmarks and a chain of summarization approach. Preprint, arXiv:2312.11865. — [arXiv:2312.11865](https://arxiv.org/abs/2312.11865) [find](https://scholar.google.com/scholar?q=Weiyu+Ma%2C+Qirui+Mi%2C+Yongcheng+Zeng%2C+Xue+Yan%2C+Yuqiao+Wu%2C+Runji+Lin%2C+Haifeng+Zhang%2C+and+Jun+Wang%2E+2024%2E+Large+language+models+play+starcraft+ii%3A+Benchmarks+and+a+chain+of+summarization+approach%2E+Preprin)
27. Zelai Xu, Zican Xu, Xuanhan Yi, Hongru Yuan, Xin Chen, Yi Wu, Chao Yu, and Yu Wang. 2025. Vs-bench: Evaluating vlms for strategic reasoning and decision-making in multi-agent environments. arXiv preprint arXiv:2506.02387. — [arXiv:2506.02387](https://arxiv.org/abs/2506.02387) [find](https://scholar.google.com/scholar?q=Zelai+Xu%2C+Zican+Xu%2C+Xuanhan+Yi%2C+Hongru+Yuan%2C+Xin+Chen%2C+Yi+Wu%2C+Chao+Yu%2C+and+Yu+Wang%2E+2025%2E+Vs%2Dbench%3A+Evaluating+vlms+for+strategic+reasoning+and+decision%2Dmaking+in+multi%2Dagent+environments%2E+arXiv+prepr)
28. Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. 2022. The surprising effectiveness of PPO in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611–24624. [find](https://scholar.google.com/scholar?q=Chao+Yu%2C+Akash+Velu%2C+Eugene+Vinitsky%2C+Jiaxuan+Gao%2C+Yu+Wang%2C+Alexandre+Bayen%2C+and+Yi+Wu%2E+2022%2E+The+surprising+effectiveness+of+PPO+in+cooperative+multi%2Dagent+games%2E+Advances+in+Neural+Information+Proce)
29. Alex L. Zhang et al. 2025a. Videogamebench: Can vision-language models complete popular video games? arXiv preprint arXiv:2505.18134. — [arXiv:2505.18134](https://arxiv.org/abs/2505.18134) [find](https://scholar.google.com/scholar?q=Alex+L%2E+Zhang+et+al%2E+2025a%2E+Videogamebench%3A+Can+vision%2Dlanguage+models+complete+popular+video+games%3F+arXiv+preprint+arXiv%3A2505%2E18134%2E+%E2%80%94+%5BarXiv%3A2505%2E18134%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2505%2E18134%29)
30. Chen Zhang, Qiang He, Zhou Yuan, Elvis S. Liu, Hong Wang, Jian Zhao, and Yang Wang. 2024. Advancing drl agents in commercial fighting games: Training, integration, and agent-human alignment. Preprint, arXiv:2406.01103. — [arXiv:2406.01103](https://arxiv.org/abs/2406.01103) [find](https://scholar.google.com/scholar?q=Chen+Zhang%2C+Qiang+He%2C+Zhou+Yuan%2C+Elvis+S%2E+Liu%2C+Hong+Wang%2C+Jian+Zhao%2C+and+Yang+Wang%2E+2024%2E+Advancing+drl+agents+in+commercial+fighting+games%3A+Training%2C+integration%2C+and+agent%2Dhuman+alignment%2E+Preprint%2C)
31. Chen Zhang, Huan Hu, Yuan Zhou, Xu Wang, and Elvis S. Liu. 2025b. Hifas: A hybrid interactive fps agent system for large game maps. IEEE Transactions on Games, pages 1–13. [find](https://scholar.google.com/scholar?q=Chen+Zhang%2C+Huan+Hu%2C+Yuan+Zhou%2C+Xu+Wang%2C+and+Elvis+S%2E+Liu%2E+2025b%2E+Hifas%3A+A+hybrid+interactive+fps+agent+system+for+large+game+maps%2E+IEEE+Transactions+on+Games%2C+pages+1%E2%80%9313%2E)
32. Zhonghan Zhao et al. 2024. Steve: See and think: Embodied agent in virtual environment. In ECCV. [find](https://scholar.google.com/scholar?q=Zhonghan+Zhao+et+al%2E+2024%2E+Steve%3A+See+and+think%3A+Embodied+agent+in+virtual+environment%2E+In+ECCV%2E)

*Updated 1 November 2026*