--- title: "Selective Multimodal Retrieval for Automated Verification of" description: "Text-based fact-checking pipelines often fail when misinformation pairs a statement with an image. 15 references." authors: ["Yoana Tsoneva", "Paul-Conrad Feig", "Jiaao Li", "Veronika Solopova", "Neda Foroutan", "Arthur Hilbert", "Vera Schmitt"] year: 2026 venue: "Proceedings of the Ninth Fact Extraction and VERification Workshop (FEVER)" source: "https://aclanthology.org/2026.fever-1.10/" doi: "10.18653/v1/2026.fever-1.10" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/tsoneva-2026-selective-multimodal-retrieval-automated-verification/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tsoneva-2026-selective-multimodal-retrieval-automated-verification/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/49/) / Selective Multimodal Retrieval for Automated Verification of Image–Text Claims # Selective Multimodal Retrieval for Automated Verification of Image–Text Claims **Authors:** Yoana Tsoneva, Paul-Conrad Feig, Jiaao Li, [Veronika Solopova](https://md.research.elamur.ai/free/en/authors/veronika-solopova/), Neda Foroutan, Arthur Hilbert and [Vera Schmitt](https://md.research.elamur.ai/free/en/authors/vera-schmitt/) · **Year:** 2026 · **Venue:** [Proceedings of the Ninth Fact Extraction and VERification Workshop (FEVER)](https://md.research.elamur.ai/free/en/venues/fever/) ([FEVER 2026](https://md.research.elamur.ai/free/en/venues/fever/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.fever-1.10/) · [PDF](https://aclanthology.org/2026.fever-1.10.pdf) · [DOI 10.18653/v1/2026.fever-1.10](https://doi.org/10.18653/v1/2026.fever-1.10) · [OpenAlex](https://openalex.org/W7140076415) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Text-based fact-checking pipelines often fail when misinformation pairs a statement with an image. Fully multimodal systems handle such claims but are expensive and can behave unpredictably. For the FEVER-9 shared task, the authors built a pipeline that decides deliberately where visual information is used, instead of passing it through every component. Qwen3-VL-8B-Instruct, used zero-shot, splits each AVerImaTeC claim into three to five verification questions and tags each one as text-related or image-related. Text questions go to a hybrid retriever: BM25 candidate selection followed by gte-base dense reranking. Image questions use SigLIP embeddings that combine the query with the referenced picture. The same model then assigns a verdict and writes a brief justification based on the collected textual evidence. The system ranked fourth overall and clearly outperformed the official baseline on evidence, verdict and justification metrics. The results suggest that targeted, modality-aware retrieval can give solid image–text verification without heavy multimodal fusion, which matters for teams with limited compute. ## Key points - **Data:** AVerImaTeC contains 1,297 real-world image–text claims, each assigned one of four labels: Refuted, Supported, Conflicting Evidence, or Not Enough Evidence. It comes with a knowledge store of scraped web text and images. - **Text retrieval:** - The store is cut into four-sentence blocks. - BM25 keeps the top 2,000 blocks. - gte-base cosine similarity selects the final 10. - **Image retrieval:** - google/siglip-base-patch16-384 embeds chunks of at most 60 tokens. - The query vector averages the text and image embeddings. - Each URL is scored by the mean of its three best chunk similarities, and the system returns 5 URLs with 3 chunks each. - **Results vs. baseline:** - Evidence Score: 0.2703 vs. 0.1707 (about 58% relative gain). - Verdict Accuracy: 0.2557 vs. 0.1136. - Justification Score: 0.198 vs. 0.1322. - Overall rank: 4th in FEVER-9. - **Limitations and code:** - The system uses one small model with zero-shot prompting only and was developed on the validation set. - Forcing questions into text/image types made them less fluent. - The effect of images on verdicts was not tested. - Code: GitHub XplaiNLP/FEVER-9-AVerImaTeC-XxP. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [German Research Centre for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/german-research-centre-for-artificial-intelligence/) — Research institute, Kaiserslautern, Germany · 79 works · 29,970 citations - [Technische Universität Berlin](https://md.research.elamur.ai/free/en/universities/technische-universitat-berlin/) — University, Berlin, Germany · 22 works · 957 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 626 works 🧭 Topics (2): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 552 works ## Cite Cite as: Yoana Tsoneva, Paul-Conrad Feig, Jiaao Li, Veronika Solopova, Neda Foroutan, Arthur Hilbert and Vera Schmitt. 2026. Selective Multimodal Retrieval for Automated Verification of Image–Text Claims. Proceedings of the Ninth Fact Extraction and VERification Workshop (FEVER). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{tsoneva2026selective, title = {{Selective Multimodal Retrieval for Automated Verification of Image–Text Claims}}, author = {Yoana Tsoneva, Paul-Conrad Feig, Jiaao Li, Veronika Solopova, Neda Foroutan, Arthur Hilbert and Vera Schmitt}, booktitle = {Proceedings of the Ninth Fact Extraction and VERification Workshop (FEVER)}, year = {2026}, doi = {10.18653/v1/2026.fever-1.10}, url = {https://doi.org/10.18653/v1/2026.fever-1.10} } ``` - [← previous Selection Shapes the Boundary: A Preregistered Replication of Monotonicity and Label Agreement in Unselected NLI Populations](https://md.research.elamur.ai/free/en/papers/choi-2026-selection-shapes-boundary-preregistered-replication/) - [next → Selective Span-Level Unlearning for Large Language Models](https://md.research.elamur.ai/free/en/papers/yoon-2026-selective-span-level-unlearning-large/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/49/) ## References 15 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Mubashara Akhtar, Michael Schlichtkrull, Zhijiang Guo, Oana Cocarascu, Elena Simperl, and Andreas Vlachos. 2023. Multimodal automated fact-checking: A survey. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5430–5448, Singapore. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Mubashara+Akhtar%2C+Michael+Schlichtkrull%2C+Zhijiang+Guo%2C+Oana+Cocarascu%2C+Elena+Simperl%2C+and+Andreas+Vlachos%2E+2023%2E+Multimodal+automated+fact%2Dchecking%3A+A+survey%2E+In+Findings+of+the+Association+for+Comput) 2. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. [Qwen3-vl technical report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen3-vl-technical-report/). Preprint, ar… 3. Rui Cao, Zifeng Ding, Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2025. Averimatec: A dataset for automatic verification of image-text claims with evidence from the web. arXiv preprint arXiv:2505.17978. — [arXiv:2505.17978](https://arxiv.org/abs/2505.17978) [find](https://scholar.google.com/scholar?q=Rui+Cao%2C+Zifeng+Ding%2C+Zhijiang+Guo%2C+Michael+Schlichtkrull%2C+and+Andreas+Vlachos%2E+2025%2E+Averimatec%3A+A+dataset+for+automatic+verification+of+image%2Dtext+claims+with+evidence+from+the+web%2E+arXiv+preprint+a) 4. Recep Firat Cekinel, Pinar Karagoz, and Çagrı Cöltekin. 2025. Multimodal fact-checking with vision language models: A probing classifier based solution with embedding strategies. In Proceedings of COLING. [find](https://scholar.google.com/scholar?q=Recep+Firat+Cekinel%2C+Pinar+Karagoz%2C+and+%C3%87agr%C4%B1+C%C3%B6ltekin%2E+2025%2E+Multimodal+fact%2Dchecking+with+vision+language+models%3A+A+probing+classifier+based+solution+with+embedding+strategies%2E+In+Proceedings+of+COL) 5. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. 2015. [Microsoft coco captions: Data collection and evaluation server](https://md.research.elamur.ai/free/en/papers/chen-2015-microsoft-coco-captions-data-collection/). Preprint, arXiv:1504.00325. 6. Michael Hameleers, Thomas E. Powell, Toni G.L.A. Van Der Meer, and Lieke Bos. 2020. A picture paints a thousand lies? the effects and mechanisms of multimodal disinformation and rebuttals disseminated via social media. Political Communication, 37(2):281– 301. [find](https://scholar.google.com/scholar?q=Michael+Hameleers%2C+Thomas+E%2E+Powell%2C+Toni+G%2EL%2EA%2E+Van+Der+Meer%2C+and+Lieke+Bos%2E+2020%2E+A+picture+paints+a+thousand+lies%3F+the+effects+and+mechanisms+of+multimodal+disinformation+and+rebuttals+disseminated) 7. Thomas Hetzner, Veronika Solopova, Vera Schmitt, and Dorothea Kolossa. 2025. Integrating video, text, and images for multimodal disinformation detection. In Proceedings of the 4th ACM International Workshop on Multimedia AI against Disinformation (MAD’25). ACM. [find](https://scholar.google.com/scholar?q=Thomas+Hetzner%2C+Veronika+Solopova%2C+Vera+Schmitt%2C+and+Dorothea+Kolossa%2E+2025%2E+Integrating+video%2C+text%2C+and+images+for+multimodal+disinformation+detection%2E+In+Proceedings+of+the+4th+ACM+International+Wo) 8. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3:333–389. [find](https://scholar.google.com/scholar?q=Stephen+Robertson+and+Hugo+Zaragoza%2E+2009%2E+The+probabilistic+relevance+framework%3A+Bm25+and+beyond%2E+Foundations+and+Trends+in+Information+Retrieval%2C+3%3A333%E2%80%93389%2E) 9. Michael Schlichtkrull, Yulong Chen, Chenxi Whitehouse, Zhenyun Deng, Mubashara Akhtar, Rami Aly, Zhijiang Guo, Christos Christodoulopoulos, Oana Cocarascu, Arpit Mittal, James Thorne, and Andreas Vlachos. 2024. The automated verification of textual claims (AVeriTeC) shared task. In Proceedings of th… [find](https://scholar.google.com/scholar?q=Michael+Schlichtkrull%2C+Yulong+Chen%2C+Chenxi+Whitehouse%2C+Zhenyun+Deng%2C+Mubashara+Akhtar%2C+Rami+Aly%2C+Zhijiang+Guo%2C+Christos+Christodoulopoulos%2C+Oana+Cocarascu%2C+Arpit+Mittal%2C+James+Thorne%2C+and+Andreas+Vlac) 10. Sahar Tahmasebi, Eric Müller-Budack, and Ralph Ewerth. 2024. Multimodal misinformation detection using large vision-language models. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, CIKM ’24, page 2189–2199, New York, NY, USA. Association for Computing… [find](https://scholar.google.com/scholar?q=Sahar+Tahmasebi%2C+Eric+M%C3%BCller%2DBudack%2C+and+Ralph+Ewerth%2E+2024%2E+Multimodal+misinformation+detection+using+large+vision%2Dlanguage+models%2E+In+Proceedings+of+the+33rd+ACM+International+Conference+on+Informat) 11. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. [Fever: a large-scale dataset for fact extraction and verification](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/). In Proceedings of NAACL-HLT. 12. Max Upravitelev, Premtim Sahitaj, Arthur Hilbert, Veronika Solopova, Jing Yang, Nils Feldhus, Tatiana Anikina, Simon Ostermann, and Vera Schmitt. 2025. Exploring semantic filtering heuristics for efficient claim verification. In Proceedings of the Eighth Fact Extraction and VERification Workshop (FE… [find](https://scholar.google.com/scholar?q=Max+Upravitelev%2C+Premtim+Sahitaj%2C+Arthur+Hilbert%2C+Veronika+Solopova%2C+Jing+Yang%2C+Nils+Feldhus%2C+Tatiana+Anikina%2C+Simon+Ostermann%2C+and+Vera+Schmitt%2E+2025%2E+Exploring+semantic+filtering+heuristics+for+effi) 13. Shengkang Wang, Hongzhan Lin, Ziyang Luo, Zhen Ye, Guang Chen, and Jing Ma. 2025. Mfc-bench: Benchmarking multimodal fact-checking with large vision-language models. In Proceedings of the Building Trust Workshop at ICLR. ArXiv:2406.11288. — [arXiv:2406.11288](https://arxiv.org/abs/2406.11288) [find](https://scholar.google.com/scholar?q=Shengkang+Wang%2C+Hongzhan+Lin%2C+Ziyang+Luo%2C+Zhen+Ye%2C+Guang+Chen%2C+and+Jing+Ma%2E+2025%2E+Mfc%2Dbench%3A+Benchmarking+multimodal+fact%2Dchecking+with+large+vision%2Dlanguage+models%2E+In+Proceedings+of+the+Building+Tru) 14. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. [Sigmoid loss for language image pre-training](https://md.research.elamur.ai/free/en/papers/zhai-2023-sigmoid-loss-language-image-pre/). In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986. 15. Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. 2024. Long-clip: Unlocking the long-text capability of clip. In European conference on computer vision, pages 310–325. Springer. [find](https://scholar.google.com/scholar?q=Beichen+Zhang%2C+Pan+Zhang%2C+Xiaoyi+Dong%2C+Yuhang+Zang%2C+and+Jiaqi+Wang%2E+2024%2E+Long%2Dclip%3A+Unlocking+the+long%2Dtext+capability+of+clip%2E+In+European+conference+on+computer+vision%2C+pages+310%E2%80%93325%2E+Springer%2E) *Updated 1 November 2026*