---
title: "A11y-Compressor — paper by Takeshita et al. (2026)"
description: "The authors propose A11y-Compressor, a modular framework that rewrites the linearized tree into a shorter, structured… 15 references."
authors: ["Michito Takeshita", "Takuro Kawada", "Takumi Ohashi", "Shunsuke Kitada", "Hitoshi Iyatomi"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)"
source: "https://aclanthology.org/2026.acl-srw.50/"
doi: "10.18653/v1/2026.acl-srw.50"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/takeshita-2026-a11y-compressor-framework-enhancing-efficiency/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/takeshita-2026-a11y-compressor-framework-enhancing-efficiency/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/20/) / A11y-Compressor: A Framework for Enhancing the Efficiency of GUI Agent Observations through Visual Context Reconstruction and Redundancy Reduction

# A11y-Compressor: A Framework for Enhancing the Efficiency of GUI Agent Observations through Visual Context Reconstruction and Redundancy Reduction

**Authors:** Michito Takeshita, Takuro Kawada, [Takumi Ohashi](https://md.research.elamur.ai/free/en/authors/takumi-ohashi/), [Shunsuke Kitada](https://md.research.elamur.ai/free/en/authors/shunsuke-kitada/) and [Hitoshi Iyatomi](https://md.research.elamur.ai/free/en/authors/hitoshi-iyatomi/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-srw.50/) · [PDF](https://aclanthology.org/2026.acl-srw.50.pdf) · [DOI 10.18653/v1/2026.acl-srw.50](https://doi.org/10.18653/v1/2026.acl-srw.50) · [OpenAlex](https://openalex.org/W7166813263)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
GUI agents built on locally hosted open-source multimodal LLMs often struggle with grounding, which means mapping on-screen elements to executable actions. The linearized accessibility (a11y) tree is a popular text observation format, but it is verbose. It also does not reflect the visual layout: pop-up dialogs sit alongside background elements, duplicates and hidden items appear, and long paragraphs inflate the context. The authors propose A11y-Compressor, a modular framework that rewrites the linearized tree into a shorter, structured observation. Their rule-based instantiation, Compressed-a11y, has three stages:
- **Modal detection** separates foreground dialogs from background content.
- **Redundancy reduction** filters irrelevant or OS-level elements, merges duplicates, trims text using the instruction, and replaces top-left boxes with center coordinates.
- **Semantic structuring** identifies the application, groups elements into functional regions, and sorts them spatially.

On the OSWorld benchmark, the method cuts input to about 22% of the original tokens and raises average task success by 5.1 percentage points. It also outperforms baselines such as LineRetriever. Cheaper and more reliable observations make local agents more practical.

## Key points
- Evaluation is on OSWorld desktop applications with local MLLMs. One example screen shrinks from 8757 to 1910 tokens.
- In the ablation, the full pipeline achieves the best overall success rate, about 0.207. Single-phase variants score roughly 0.13–0.16.
- Redundancy reduction is the strongest single phase, which the authors attribute to center-based click coordinates. Semantic structuring alone works best in Calc.
- LineRetriever, which selects lines with an LLM, produces compact input but loses global UI structure, so its gains are unstable.
- Limitations:
  - The method uses no purely visual cues such as icons or colors.
  - It has not been tested on mobile interfaces.
  - It relies on heuristic, rule-based thresholds.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Hosei University](https://md.research.elamur.ai/free/en/universities/hosei-university/) — University, Tokyo, Japan · 3 works · 4 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works · [🖱️ Human-Computer Interaction](https://md.research.elamur.ai/free/en/shelves/computer-science/human-computer-interaction/) · 141 works

## Cite

Cite as: Michito Takeshita, Takuro Kawada, Takumi Ohashi, Shunsuke Kitada and Hitoshi Iyatomi. 2026. A11y-Compressor: A Framework for Enhancing the Efficiency of GUI Agent Observations through Visual Context Reconstruction and Redundancy Reduction. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{takeshita2026a11y,
  title = {{A11y-Compressor: A Framework for Enhancing the Efficiency of GUI Agent Observations through Visual Context Reconstruction and Redundancy Reduction}},
  author = {Michito Takeshita, Takuro Kawada, Takumi Ohashi, Shunsuke Kitada and Hitoshi Iyatomi},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-srw.50},
  url = {https://doi.org/10.18653/v1/2026.acl-srw.50}
}
```

- [← previous A Zipfian Analysis of Visual Token Distributions for AI-Generated Images](https://md.research.elamur.ai/free/en/papers/shin-2026-zipfian-analysis-visual-token-distributions/)
- [next → A2O: LLM-based Agentic Learning of Action-to-Object Features for Video Action Recognition](https://md.research.elamur.ai/free/en/papers/sekii-2026-a2o-llm-based-agentic-learning/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/20/)

## References

15 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. [Qwen3-VL Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen3-vl-technical-report/). [https://doi.…](https://doi.…) [↗](https://doi.…)
2. Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://md.research.elamur.ai/free/en/papers/cheng-2024-seeclick-harnessing-gui-grounding-advanced/). In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
3. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Boshi Stevens, Samuel andus, Huan Sun, and Yu Su. 2023. [Mind2Web: Towards a Generalist Agent for the Web](https://md.research.elamur.ai/free/en/papers/deng-2023-mind2web-towards-generalist-agent-web/). In Advances in Neural Information Processing Systems (NeurIPS).
4. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. 2024. [CogAgent: A Visual Language Model for GUI Agents](https://md.research.elamur.ai/free/en/papers/hong-2023-cogagent-visual-language-model/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco…
5. Imene Kerboua, Sahar Omidi Shayegan, Megh Thakkar, Xing Han Lù, Massimo Caccia, Véronique Eglin, Alexandre Aussem, Jérémy Espinas, and Alexandre Lacoste. 2025. LineRetriever: Planning-Aware Observation Reduction for Web Agents. https://doi.org/10.48550/arXiv.2507.00210. — [doi:10.48550/arxiv.2507.00210](https://doi.org/10.48550/arxiv.2507.00210) [↗](https://doi.org/10.48550/arXiv.2507.00210)
6. Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Stan Weixian Lei, Lijuan Wang, and Mike Zheng Shou. 2025. ShowUI: One Vision-Language-Action Model for GUI Visual Agent. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [find](https://scholar.google.com/scholar?q=Kevin+Qinghong+Lin%2C+Linjie+Li%2C+Difei+Gao%2C+Zhengyuan+Yang%2C+Shiwei+Wu%2C+Zechen+Bai%2C+Stan+Weixian+Lei%2C+Lijuan+Wang%2C+and+Mike+Zheng+Shou%2E+2025%2E+ShowUI%3A+One+Vision%2DLanguage%2DAction+Model+for+GUI+Visual+Agent)
7. Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang. 2024. ScreenAgent: A Vision Language Model-driven Computer Control Agent. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI). [find](https://scholar.google.com/scholar?q=Runliang+Niu%2C+Jindong+Li%2C+Shiqi+Wang%2C+Yali+Fu%2C+Xiyu+Hu%2C+Xueyuan+Leng%2C+He+Kong%2C+Yi+Chang%2C+and+Qi+Wang%2E+2024%2E+ScreenAgent%3A+A+Vision+Language+Model%2Ddriven+Computer+Control+Agent%2E+In+Proceedings+of+the+Th)
8. Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024. Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration. In Advances in Neural Information Processing Systems (NeurIPS). [find](https://scholar.google.com/scholar?q=Junyang+Wang%2C+Haiyang+Xu%2C+Haitao+Jia%2C+Xi+Zhang%2C+Ming+Yan%2C+Weizhou+Shen%2C+Ji+Zhang%2C+Fei+Huang%2C+and+Jitao+Sang%2E+2024%2E+Mobile%2DAgent%2Dv2%3A+Mobile+Device+Operation+Assistant+with+Effective+Navigation+via+Mult)
9. Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang, Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao, Reuben Tan, Si Qin, Lars Liden, Qingwei Lin, Huan Zhang, Tong Zhang, Jianbing Zhang, Dongmei Zhang, and Jianfeng Gao. 2025. GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents. In A… [find](https://scholar.google.com/scholar?q=Qianhui+Wu%2C+Kanzhi+Cheng%2C+Rui+Yang%2C+Chaoyun+Zhang%2C+Jianwei+Yang%2C+Huiqiang+Jiang%2C+Jian+Mu%2C+Baolin+Peng%2C+Bo+Qiao%2C+Reuben+Tan%2C+Si+Qin%2C+Lars+Liden%2C+Qingwei+Lin%2C+Huan+Zhang%2C+Tong+Zhang%2C+Jianbing+Zhang%2C+Don)
10. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. 2024. [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in…](https://md.research.elamur.ai/free/en/papers/xie-2024-osworld-benchmarking-multimodal-agents-open/)
11. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. [Qwen3 Technical Report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). https://doi.org/10.48550/arXiv.25… [↗](https://doi.org/10.48550/arXiv.25…)
12. Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. 2025a. [UFO: A UI-Focused Agent for Windows OS Interaction](https://md.research.elamur.ai/free/en/papers/zhang-2025-ufo-ui-focused-agent-windows/). In Proceedings of the Conference of the Nations of the Americas Chapter of the Association for Co…
13. Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Han. 2025b. AppAgent: Multimodal Agents as Smartphone Users. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (CHI). [find](https://scholar.google.com/scholar?q=Chi+Zhang%2C+Zhao+Yang%2C+Jiaxuan+Liu%2C+Yucheng+Han%2C+Xin+Chen%2C+Zebiao+Huang%2C+Bin+Fu%2C+and+Gang+Han%2E+2025b%2E+AppAgent%3A+Multimodal+Agents+as+Smartphone+Users%2E+In+Proceedings+of+the+ACM+CHI+Conference+on+Human+)
14. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. [GPT-4V(ision) Is a Generalist Web Agent, If Grounded](https://md.research.elamur.ai/free/en/papers/zheng-2024-gpt-4v-ision-generalist-web/). In Proceedings of the 41st International Conference on Machine Learning (ICML).
15. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xian Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://md.research.elamur.ai/free/en/papers/zhou-2023-webarena-realistic-web-environment-building/). In Proceedings of the International Conference on Learning Representations (ICLR)…

*Updated 1 November 2026*