--- title: "Spectra — paper by Neo et al. (2026) — summary and ideas" description: "The authors present Spectra, an MIT-licensed Python library built on HuggingFace Transformers. 4 references." authors: ["Clement Neo", "Yongsen Zheng", "Kwok-Yan Lam", "Luke Ong"] year: 2026 venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)" source: "https://aclanthology.org/2026.acl-demo.78/" doi: "10.18653/v1/2026.acl-demo.78" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/neo-2026-spectra-mechanistic-interpretability-library-vision/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/neo-2026-spectra-mechanistic-interpretability-library-vision/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/50/) / Spectra: A Mechanistic Interpretability Library for Vision-Language Models # Spectra: A Mechanistic Interpretability Library for Vision-Language Models **Authors:** Clement Neo, Yongsen Zheng, [Kwok-Yan Lam](https://md.research.elamur.ai/free/en/authors/kwok-yan-lam/) and Luke Ong · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.acl-demo.78/) · [PDF](https://aclanthology.org/2026.acl-demo.78.pdf) · [DOI 10.18653/v1/2026.acl-demo.78](https://doi.org/10.18653/v1/2026.acl-demo.78) · [OpenAlex](https://openalex.org/W7166889655) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Tooling for mechanistic interpretability of vision-language models (VLMs) lags behind text-only tooling such as TransformerLens. Existing VLM tools like vlm-lens mostly record activations and cannot intervene in the model. The difficulty is that adapter-style VLMs differ in how they mark image tokens, how they arrange them, and how their adapters merge patches. The authors present Spectra, an MIT-licensed Python library built on HuggingFace Transformers. Its HookedVLM wrapper provides context managers for caching activations and running hooked interventions. Per-checkpoint adapter configurations hide model-specific details. The library also offers helpers for locating image tokens, tracking crop and resize metadata, and overlaying the patch grid on an image. A logit lens and a web app are included. As a case study, the authors analyze object counting on CountBench using ablation, attention blocking and layer-wise activation patching. Benchmarks show little overhead for caching hidden states. This matters because researchers can run causal VLM experiments without writing fragile per-model indexing code. ## Key points - Supported models: Qwen2.5-VL, Qwen3-VL, LLaVA 1.5 and SmolVLM. Hook names largely follow TransformerLens conventions. The code is at github.com/clemneo/vlm-spectra. - Counting experiments: - Zeroing count-number positions at the language model input reduced their average number from 30.8 to 11.8. - Blocking attention to count tokens changed the answer for only 4 of 100 images. - Activation patching (2 vs. 7 children): image-only and text-only patching cross over around layers 15–25. Restricting patching to the final 16 or 32 tokens of the image yields a nearly negligible effect, which suggests the count information is spread across many tokens. - Testing has three tiers: unit tests, integration tests on SmolVLM-256M, and per-model acceptance tests. Cached activations are checked against HuggingFace's native outputs. - Overhead on 300 CountBench images with LLaVA-1.5-7B: - Caching hidden states adds about +2.1% (24.6 s vs. 24.0 s for a plain forward pass). - Caching attention patterns adds about 15%. - Patching attention heads adds about 72%. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Nanyang Technological University](https://md.research.elamur.ai/free/en/universities/nanyang-technological-university/) — University, Singapore, Singapore · 184 works · 27,484 citations 📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works 🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works · [🔬 Interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/) · 149 works ## Cite Cite as: Clement Neo, Yongsen Zheng, Kwok-Yan Lam and Luke Ong. 2026. Spectra: A Mechanistic Interpretability Library for Vision-Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{neo2026spectra, title = {{Spectra: A Mechanistic Interpretability Library for Vision-Language Models}}, author = {Clement Neo, Yongsen Zheng, Kwok-Yan Lam and Luke Ong}, booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)}, year = {2026}, doi = {10.18653/v1/2026.acl-demo.78}, url = {https://doi.org/10.18653/v1/2026.acl-demo.78} } ``` - [← previous Specialization without Sparsity: Efficient and Expressive Split-Path Experts for LLM Fine-Tuning](https://md.research.elamur.ai/free/en/papers/liu-2026-specialization-without-sparsity-efficient-expressive/) - [next → Spectral Characterization and Mitigation of Sequential Knowledge Editing Collapse](https://md.research.elamur.ai/free/en/papers/zhang-2026-spectral-characterization-mitigation-sequential-knowledge/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/50/) ## References 4 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Estelle Aflalo, Meng Du, Shao-Yen Tseng, Yongfei Liu, Chenfei Wu, Nan Duan, and Vasudev Lal. 2022. Vl-interpret: An interactive visualization tool for interpreting vision-language transformers. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 21406–21415. [find](https://scholar.google.com/scholar?q=Estelle+Aflalo%2C+Meng+Du%2C+Shao%2DYen+Tseng%2C+Yongfei+Liu%2C+Chenfei+Wu%2C+Nan+Duan%2C+and+Vasudev+Lal%2E+2022%2E+Vl%2Dinterpret%3A+An+interactive+visualization+tool+for+interpreting+vision%2Dlanguage+transformers%2E+In+Pro) 2. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025a. [Qwen3-vl technical report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen3-vl-technical-report/). Preprint, a… 3. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025b. [Qwen2.5-vl technical report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen2-5-vl-technical/). Preprint, arXiv:2502.1… 4. Caching attention patterns HuggingFace 26.7 +11.1% Spectra 30.9 +28.4% Activation intervention (run_with_hooks) Residual-stream patch 25.6 +6.4% Head patch 41.4 +72.3% Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do n… [find](https://scholar.google.com/scholar?q=Caching+attention+patterns+HuggingFace+26%2E7+%2B11%2E1%25+Spectra+30%2E9+%2B28%2E4%25+Activation+intervention+%28run%5Fwith%5Fhooks%29+Residual%2Dstream+patch+25%2E6+%2B6%2E4%25+Head+patch+41%2E4+%2B72%2E3%25+Initiative%2E+Any+opinions%2C+findin) *Updated 1 November 2026*