---
title: "On the Additive Compositionality of Task Vectors in"
description: "They build compositional visual reasoning tasks from CLEVR (five single-attribute subtasks: shape, color, material, size… 22 references."
authors: ["Yuting Shi", "Houjing Wei", "Naoya Inoue"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/2026.eacl-short.38/"
doi: "10.18653/v1/2026.eacl-short.38"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shi-2026-additive-compositionality-task-vectors-vision/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shi-2026-additive-compositionality-task-vectors-vision/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/38/) / On the Additive Compositionality of Task Vectors in Vision–Language Models

# On the Additive Compositionality of Task Vectors in Vision–Language Models

**Authors:** Yuting Shi, Houjing Wei and [Naoya Inoue](https://md.research.elamur.ai/free/en/authors/naoya-inoue/) · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-short.38/) · [PDF](https://aclanthology.org/2026.eacl-short.38.pdf) · [DOI 10.18653/v1/2026.eacl-short.38](https://doi.org/10.18653/v1/2026.eacl-short.38) · [OpenAlex](https://openalex.org/W7140080303)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The authors test whether task-vector arithmetic, known from text-only LLMs, also works in vision–language models, where visual encoders and cross-modal alignment could break linear behaviour. They build compositional visual reasoning tasks from CLEVR (five single-attribute subtasks: shape, color, material, size, position, giving ten pairs) and GQA (three attributes, three pairs). A task vector is the hidden state of the last text token at a chosen layer, read from a prompt that contains demonstrations and a fixed dummy query. This vector is then added at the same layer to a prompt that has only the query and no demonstrations. Experiments use IDEFICS-8B, LLaVA-1.5-7B and Qwen2-VL-7B-Instruct. Summing two single-attribute vectors performs about as well as a vector taken directly from the combined task, and much better than random sums. Attention-head ablations and token-level embedding comparisons suggest that complex-task representations are superpositions of their atomic parts. This makes in-context behaviour in VLMs more interpretable and modular.

## Key points
- **Setup:** Evaluation uses 200 test instances per task, averaged over three random seeds. The best injection layers are mostly 17 or 20 for IDEFICS-8B.
- **Injection quality and model differences:** Injecting a subtask vector without any demonstrations closely reproduces 3-shot accuracy. IDEFICS-8B shows the strongest additive approximation, Qwen2-VL-7B shows moderate approximation, and LLaVA-1.5-7B shows the weakest. These differences track each model's zero-shot and few-shot ability.
- **Three-attribute tasks:**
  - For color-size-material, the summed vector scores 0.96, versus 0.86 for the directly extracted vector and 0.58 for random sums.
  - For shape-color-size, the scores are 0.90 versus 0.87.
  - Tasks that include position compose less reliably (shape-color-position: 0.52 versus 0.77).
- **Shared mechanisms:** The ten most important attention heads, ranked by an ablation-based ΔCIE score, overlap by about 0.7 between composed-vector injection and complex-vector injection.
- **Token-level evidence and limits:**
  - Identical tokens in atomic and complex prompts (color–size task) reach cosine similarity near 0.9, versus about 0.3 for mismatched tokens.
  - The study is limited to controlled tasks and simple vector addition.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Japan Advanced Institute of Science and Technology](https://md.research.elamur.ai/free/en/universities/japan-advanced-institute-of-science-and-technology/) — University, Nomi Shi, Japan · 7 works · 980 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works

## Cite

Cite as: Yuting Shi, Houjing Wei and Naoya Inoue. 2026. On the Additive Compositionality of Task Vectors in Vision–Language Models. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shi2026additive,
  title = {{On the Additive Compositionality of Task Vectors in Vision–Language Models}},
  author = {Yuting Shi, Houjing Wei and Naoya Inoue},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-short.38},
  url = {https://doi.org/10.18653/v1/2026.eacl-short.38}
}
```

- [← previous On Temperature-Constrained Non-Deterministic Machine Translation: Potential and Evaluation](https://md.research.elamur.ai/free/en/papers/wang-2026-temperature-constrained-non-deterministic-machine/)
- [next → On the Cultural Anachronism and Temporal Reasoning in Vision Language Models](https://md.research.elamur.ai/free/en/papers/ranjan-2026-cultural-anachronism-temporal-reasoning-vision/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/38/)

## References

22 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Davide Berasi, Matteo Farina, Massimiliano Mancini, Elisa Ricci, and Nicola Strisciuglio. 2025. Not only text: Exploring compositionality of visual representations in vision-language models. arXiv preprint arXiv:2503.17142. — [arXiv:2503.17142](https://arxiv.org/abs/2503.17142) [find](https://scholar.google.com/scholar?q=Davide+Berasi%2C+Matteo+Farina%2C+Massimiliano+Mancini%2C+Elisa+Ricci%2C+and+Nicola+Strisciuglio%2E+2025%2E+Not+only+text%3A+Exploring+compositionality+of+visual+representations+in+vision%2Dlanguage+models%2E+arXiv+pre)
2. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2023. [Instructblip: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/). Advances in neural information processing systems, 36:49250–49267.
3. Roee Hendel, Mor Geva, and Amir Globerson. 2023. [In-context learning creates task vectors](https://md.research.elamur.ai/free/en/papers/hendel-2023-context-learning-creates-task-vectors/). arXiv preprint arXiv:2310.15916.
4. Brandon Huang, Chancharik Mitra, Assaf Arbelle, Leonid Karlinsky, Trevor Darrell, and Roei Herzig. 2024a. Multimodal task vectors enable many-shot multimodal in-context learning. Preprint, arXiv:2406.15334. Published in NeurIPS 2024. [find](https://scholar.google.com/scholar?q=Brandon+Huang%2C+Chancharik+Mitra%2C+Assaf+Arbelle%2C+Leonid+Karlinsky%2C+Trevor+Darrell%2C+and+Roei+Herzig%2E+2024a%2E+Multimodal+task+vectors+enable+many%2Dshot+multimodal+in%2Dcontext+learning%2E+Preprint%2C+arXiv%3A2406%2E)
5. Irene Huang, Wei Lin, Muhammad Jehanzeb Mirza, Jacob Hansen, Sivan Doveh, Victor Butoi, Roei Herzig, Assaf Arbelle, Hilde Kuehne, Trevor Darrell, and 1 others. 2024b. Conme: Rethinking evaluation of compositional reasoning for modern vlms. Advances in Neural Information Processing Systems, 37:22927–… [find](https://scholar.google.com/scholar?q=Irene+Huang%2C+Wei+Lin%2C+Muhammad+Jehanzeb+Mirza%2C+Jacob+Hansen%2C+Sivan+Doveh%2C+Victor+Butoi%2C+Roei+Herzig%2C+Assaf+Arbelle%2C+Hilde+Kuehne%2C+Trevor+Darrell%2C+and+1+others%2E+2024b%2E+Conme%3A+Rethinking+evaluation+of+c)
6. Drew A Hudson and Christopher D Manning. 2019. [Gqa: A new dataset for real-world visual reasoning and compositional question answering](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/). In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709.
7. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. [Editing models with task arithmetic](https://md.research.elamur.ai/free/en/papers/ilharco-2022-editing-models-task-arithmetic/). In International Conference on Learning Representations (ICLR) 2023. ArXiv preprint arXiv:2212.04089.
8. Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. [Clevr: A diagnostic dataset for compositional language and elementary visual reasoning](https://md.research.elamur.ai/free/en/papers/johnson-2016-clevr-diagnostic-dataset-compositional-language/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901–291…
9. Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. 2023a. [Obelics: An open web-scale filtered dataset of interleaved image-text documents](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2023-obelics-open-web-scale-filtered/). Preprint, arXiv:2306.…
10. Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. 2023b. [Obelics: An open web-scale filtered dataset of interleaved image-text documents](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2023-obelics-open-web-scale-filtered/). Preprint, arXiv:2306.…
11. Prompt algebra for task composition. arXiv preprint arXiv:2306.00310. — [arXiv:2306.00310](https://arxiv.org/abs/2306.00310) [find](https://scholar.google.com/scholar?q=Prompt+algebra+for+task+composition%2E+arXiv+preprint+arXiv%3A2306%2E00310%2E+%E2%80%94+%5BarXiv%3A2306%2E00310%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2306%2E00310%29)
12. Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019. [A corpus for reasoning about natural language grounded in photographs](https://md.research.elamur.ai/free/en/papers/suhr-2019-corpus-reasoning-natural-language-grounded/). Preprint, arXiv:1811.00491.
13. Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. 2023. Function vectors in large language models. arXiv preprint arXiv:2310.15213. [find](https://scholar.google.com/scholar?q=Eric+Todd%2C+Millicent+L+Li%2C+Arnab+Sen+Sharma%2C+Aaron+Mueller%2C+Byron+C+Wallace%2C+and+David+Bau%2E+2023%2E+Function+vectors+in+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E15213%2E)
14. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. [Qwen2-vl: Enhancing vision-language model’s perception of the w…](https://md.research.elamur.ai/free/en/papers/wang-2024-qwen2-vl-enhancing-vision/)
15. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). Preprint, arXiv:22…
16. Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022. [An explanation of in-context learning as implicit bayesian inference](https://md.research.elamur.ai/free/en/papers/xie-2021-explanation-context-learning-implicit-bayesian/). Preprint, arXiv:2111.02080.
17. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023a. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). Advances in neural information processing systems, 36:34892– 34916.
18. Sheng Liu, Haotian Ye, Lei Xing, and James Zou. 2023b. In-context vectors: Making in context learning more effective and controllable through latent space steering. arXiv preprint arXiv:2311.06668. [find](https://scholar.google.com/scholar?q=Sheng+Liu%2C+Haotian+Ye%2C+Lei+Xing%2C+and+James+Zou%2E+2023b%2E+In%2Dcontext+vectors%3A+Making+in+context+learning+more+effective+and+controllable+through+latent+space+steering%2E+arXiv+preprint+arXiv%3A2311%2E06668%2E)
19. Grace Luo, Trevor Darrell, and Amir Bar. 2024. Vision-language models create cross-modal task representations. arXiv preprint arXiv:2410.22330. [find](https://scholar.google.com/scholar?q=Grace+Luo%2C+Trevor+Darrell%2C+and+Amir+Bar%2E+2024%2E+Vision%2Dlanguage+models+create+cross%2Dmodal+task+representations%2E+arXiv+preprint+arXiv%3A2410%2E22330%2E)
20. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. [Rethinking the role of demonstrations: What makes in-context learning work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) Preprint, arXiv:2202.12837.
21. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, and 7 others. 2022. [In-con…](https://md.research.elamur.ai/free/en/papers/olsson-2022-context-learning-induction-heads/)
22. Pramuditha Perera, Matthew Trager, Luca Zancato, Alessandro Achille, and Stefano Soatto. 2023. [find](https://scholar.google.com/scholar?q=Pramuditha+Perera%2C+Matthew+Trager%2C+Luca+Zancato%2C+Alessandro+Achille%2C+and+Stefano+Soatto%2E+2023%2E)

*Updated 1 November 2026*