---
title: "wenbin@EEUCA 2026 — paper by Shen (2026)"
description: "Classifying the stance of vaccine-related memes is difficult because their meaning often depends on irony, metaphor, or a… 28 references."
authors: ["Wenbin Shen"]
year: 2026
venue: "Proceedings of the 9th Workshop on Event Extraction and Understanding: Challenges and Applications (EEUCA 2026)"
source: "https://aclanthology.org/2026.eeuca-1.15/"
doi: "10.18653/v1/2026.eeuca-1.15"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shen-2026-wenbin-eeuca-2026-moes-vaxagent/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shen-2026-wenbin-eeuca-2026-moes-vaxagent/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/56/) / wenbin@EEUCA 2026: MoEs-VaxAgent, A Two-Stage Framework for Multimodal Vaccine Critical Meme Detection

# wenbin@EEUCA 2026: MoEs-VaxAgent, A Two-Stage Framework for Multimodal Vaccine Critical Meme Detection

**Authors:** Wenbin Shen · **Year:** 2026 · **Venue:** [Proceedings of the 9th Workshop on Event Extraction and Understanding: Challenges and Applications (EEUCA 2026)](https://md.research.elamur.ai/free/en/venues/eeuca/) ([Workshop on Event Extraction and Understanding: Challenges and Applications 2026](https://md.research.elamur.ai/free/en/venues/eeuca/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eeuca-1.15/) · [PDF](https://aclanthology.org/2026.eeuca-1.15.pdf) · [DOI 10.18653/v1/2026.eeuca-1.15](https://doi.org/10.18653/v1/2026.eeuca-1.15) · [OpenAlex](https://openalex.org/W7166808926)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Classifying the stance of vaccine-related memes is difficult because their meaning often depends on irony, metaphor, or a deliberate mismatch between the image and its caption. This system paper for the EEUCA 2026 VaxMeme shared task tackles a three-way labelling problem (Pro-vaccine, Vaccine-critical, Neutral), ranked by Macro F1. The proposed MoEs-VaxAgent pairs a cheap classifier with expensive LLM reasoning, which is used only where necessary.

In stage one, five MLP experts receive features from four sources:
- RoBERTa,
- ViT,
- CLIP (both text and vision encoders),
- Sentence-BERT embeddings of background text drawn from MM-CoVaR.

A learned gate selects two experts per meme and blends their outputs for prediction. In stage two, predictions below a confidence threshold of 0.5 are re-examined. An LLM text agent and a visual agent each give a verdict, and a judge agent resolves cases where they disagree.

The system reached Macro F1 0.8205 and placed 9th. The paper is valuable as a practical template for budget-aware hybrid classification. It also documents many tweaks that did not help.

## Key points
- **Data:** VaxMeme contains over 10,000 Twitter memes, split into 8,195 train, 1,024 validation and 1,025 test samples. External context comes from MM-CoVaR, which covers 2,593 news articles and 24,184 tweets from Feb 2020 to Mar 2021.
- **Experts and gating:** The Text expert uses RoBERTa+CLIP-text, the Image expert ViT+CLIP-vision, the Cross-Modal expert RoBERTa+ViT, the Alignment/CLIP expert both CLIP encoders, and the Context expert Sentence-BERT. Routing uses Top-k gating with k = 2.
- **Leaderboard results:** The system scored Macro F1 0.8205 and accuracy 0.8244, ranking 9th. The winning team scored 0.8494.
- **Cost of stage two:** On the validation set, about 14.5% of samples (149) were sent to the agents. In 76 of these the two agents agreed, and 73 went to the judge. The paper gives the validation size as 1,025 in this analysis, unlike its split table.
- **Errors and unsuccessful strategies:**
  - The main confusion is between the Vaccine-critical and Neutral classes.
  - Wrong predictions made with high confidence skip refinement entirely.
  - Reliance on LLM APIs limits the system to offline analysis.
  - Approaches that did not help include focal and contrastive losses, pseudo-labelling, one-vs-rest decomposition and XGBoost/KNN/Random Forest heads.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Nanjing University of Science and Technology](https://md.research.elamur.ai/free/en/universities/nanjing-university-of-science-and-technology/) — University, Nanjing, China · 19 works · 1,590 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 626 works

🧭 Topics (1): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works

## Cite

Cite as: Wenbin Shen. 2026. wenbin@EEUCA 2026: MoEs-VaxAgent, A Two-Stage Framework for Multimodal Vaccine Critical Meme Detection. Proceedings of the 9th Workshop on Event Extraction and Understanding: Challenges and Applications (EEUCA 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shen2026wenbin,
  title = {{wenbin@EEUCA 2026: MoEs-VaxAgent, A Two-Stage Framework for Multimodal Vaccine Critical Meme Detection}},
  author = {Wenbin Shen},
  booktitle = {Proceedings of the 9th Workshop on Event Extraction and Understanding: Challenges and Applications (EEUCA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.eeuca-1.15},
  url = {https://doi.org/10.18653/v1/2026.eeuca-1.15}
}
```

- [← previous WeightLoRA: Keep Only Necessary Adapters](https://md.research.elamur.ai/free/en/papers/veprikov-2026-weightlora-keep-only-necessary-adapters/)
- [next → WenetSpeech-Wu: Datasets, Benchmarks, and Models for a Unified Chinese Wu Dialect Speech Processing Ecosystem](https://md.research.elamur.ai/free/en/papers/wang-2026-wenetspeech-wu-datasets-benchmarks-models/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/56/)

## References

28 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aashish Bhandari, Siddhant B Shah, Surendrabikram Thapa, Usman Naseem, and Mehwish Nasim. 2023. Crisishatemm: Multimodal analysis of directed and undirected hate speech in text-embedded images from russia-ukraine conflict. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco… [find](https://scholar.google.com/scholar?q=Aashish+Bhandari%2C+Siddhant+B+Shah%2C+Surendrabikram+Thapa%2C+Usman+Naseem%2C+and+Mehwish+Nasim%2E+2023%2E+Crisishatemm%3A+Multimodal+analysis+of+directed+and+undirected+hate+speech+in+text%2Dembedded+images+from+ru)
2. Mingxuan Chen, Xinqiao Chu, and KP Subbalakshmi. 2021. Mmcovar: multimodal covid-19 vaccine focused data repository for fake news detection and a baseline architecture for classification. In Proceedings of the 2021 IEEE/ACM international conference on advances in social networks analysis and mining,… [find](https://scholar.google.com/scholar?q=Mingxuan+Chen%2C+Xinqiao+Chu%2C+and+KP+Subbalakshmi%2E+2021%2E+Mmcovar%3A+multimodal+covid%2D19+vaccine+focused+data+repository+for+fake+news+detection+and+a+baseline+architecture+for+classification%2E+In+Proceedin)
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies,…
4. Alexey Dosovitskiy. 2020. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). arXiv preprint arXiv:2010.11929.
5. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2023. [Improving factuality and reasoning in language models through multiagent debate](https://md.research.elamur.ai/free/en/papers/du-2023-improving-factuality-reasoning-language-models/). In Forty-first International Conference on Machine Learning.
6. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778.
7. Ali Hürriyetoğlu, Surendrabikram Thapa, Hristo Tanev, Laxmi Thapa, and Surabhi Adhikari. 2026. Overview of the workshop on event extraction and understanding: Challenges and applications. In Proceedings of the 9th Workshop on Event Extraction and Understanding: Challenges and Applications (EEUCA). [find](https://scholar.google.com/scholar?q=Ali+H%C3%BCrriyetog%CC%86lu%2C+Surendrabikram+Thapa%2C+Hristo+Tanev%2C+Laxmi+Thapa%2C+and+Surabhi+Adhikari%2E+2026%2E+Overview+of+the+workshop+on+event+extraction+and+understanding%3A+Challenges+and+applications%2E+In+Proceedi)
8. EunJeong Hwang and Vered Shwartz. 2023. MemeCap: A dataset for captioning and interpreting memes. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1433–1445, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=EunJeong+Hwang+and+Vered+Shwartz%2E+2023%2E+MemeCap%3A+A+dataset+for+captioning+and+interpreting+memes%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1433%E2%80%93)
9. Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020. [The hateful memes challenge: Detecting hate speech in multimodal memes](https://md.research.elamur.ai/free/en/papers/kiela-2020-hateful-memes-challenge-detecting-hate/). Advances in neural information processing systems, 33:2611–2624.
10. Hongzhan Lin, Ziyang Luo, Wei Gao, Jing Ma, Bo Wang, and Ruichao Yang. 2024. Towards explainable harmful meme detection through multimodal debate between large language models. In Proceedings of the ACM Web Conference 2024, pages 2359–2370. [find](https://scholar.google.com/scholar?q=Hongzhan+Lin%2C+Ziyang+Luo%2C+Wei+Gao%2C+Jing+Ma%2C+Bo+Wang%2C+and+Ruichao+Yang%2E+2024%2E+Towards+explainable+harmful+meme+detection+through+multimodal+debate+between+large+language+models%2E+In+Proceedings+of+the+A)
11. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692.
12. Ziyan Liu, Chunxiao Fan, Haoran Lou, Yuexin Wu, and Kaiwei Deng. 2025. MIND: A multi-agent framework for zero-shot harmful meme detection. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 923–947, Vienna, Austria. Association f… [find](https://scholar.google.com/scholar?q=Ziyan+Liu%2C+Chunxiao+Fan%2C+Haoran+Lou%2C+Yuexin+Wu%2C+and+Kaiwei+Deng%2E+2025%2E+MIND%3A+A+multi%2Dagent+framework+for+zero%2Dshot+harmful+meme+detection%2E+In+Proceedings+of+the+63rd+Annual+Meeting+of+the+Association+)
13. Fernando Andres Lovera, Yudith Coromoto Cardinale, and Masun Nabhan Homsi. 2021. Sentiment analysis in twitter based on knowledge graph and deep learning classification. Electronics, 10(22):2739. [find](https://scholar.google.com/scholar?q=Fernando+Andres+Lovera%2C+Yudith+Coromoto+Cardinale%2C+and+Masun+Nabhan+Homsi%2E+2021%2E+Sentiment+analysis+in+twitter+based+on+knowledge+graph+and+deep+learning+classification%2E+Electronics%2C+10%2822%29%3A2739%2E)
14. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2023. Self-refine: Iterative refinement with self-feedback, 2023. URL https://arxiv.org/abs/2303.17651. [↗](https://arxiv.org/abs/2303.17651)
15. Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. 2022. Multimodal contrastive learning with limoe: the language-image mixture of experts. Advances in Neural Information Processing Systems, 35:9564–9576. [find](https://scholar.google.com/scholar?q=Basil+Mustafa%2C+Carlos+Riquelme%2C+Joan+Puigcerver%2C+Rodolphe+Jenatton%2C+and+Neil+Houlsby%2E+2022%2E+Multimodal+contrastive+learning+with+limoe%3A+the+language%2Dimage+mixture+of+experts%2E+Advances+in+Neural+Inform)
16. Usman Naseem, Adam G Dunn, Matloob Khushi, and Jinman Kim. 2024. Vaccine misinformation detection in x using cooperative multimodal framework. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 4034–4042. [find](https://scholar.google.com/scholar?q=Usman+Naseem%2C+Adam+G+Dunn%2C+Matloob+Khushi%2C+and+Jinman+Kim%2E+2024%2E+Vaccine+misinformation+detection+in+x+using+cooperative+multimodal+framework%2E+In+Proceedings+of+the+32nd+ACM+International+Conference+o)
17. Usman Naseem, Jinman Kim, Matloob Khushi, and Adam G Dunn. 2023. A multimodal framework for the identification of vaccine critical memes on twitter. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, pages 706–714. [find](https://scholar.google.com/scholar?q=Usman+Naseem%2C+Jinman+Kim%2C+Matloob+Khushi%2C+and+Adam+G+Dunn%2E+2023%2E+A+multimodal+framework+for+the+identification+of+vaccine+critical+memes+on+twitter%2E+In+Proceedings+of+the+Sixteenth+ACM+International+C)
18. Shraman Pramanick, Shivam Sharma, Dimitar Dimitrov, Md. Shad Akhtar, Preslav Nakov, and Tanmoy Chakraborty. 2021. [MOMENTA: A multimodal framework for detecting harmful memes and their targets](https://md.research.elamur.ai/free/en/papers/pramanick-2021-momenta-multimodal-framework-detecting-harmful/). In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4439–4455, Punta Cana, Domi…
19. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In International conference on machine learning, pages 8748–8…
20. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2C+Azalia+Mirhoseini%2C+Krzysztof+Maziarz%2C+Andy+Davis%2C+Quoc+Le%2C+Geoffrey+Hinton%2C+and+Jeff+Dean%2E+2017%2E+Outrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2E+arXiv+pr)
21. Laxmi Thapa, Aryaman Jain, Lakshmojee Koduru, Surabhi Adhikari, Junaid Rashid, Jungeun Kim, Surendrabikram Thapa, and Usman Naseem. 2026a. Concept-grounded detection of vaccine misinformation in multimodal content using interpretable vision-language models. In Companion Proceedings of the ACM on Web… [find](https://scholar.google.com/scholar?q=Laxmi+Thapa%2C+Aryaman+Jain%2C+Lakshmojee+Koduru%2C+Surabhi+Adhikari%2C+Junaid+Rashid%2C+Jungeun+Kim%2C+Surendrabikram+Thapa%2C+and+Usman+Naseem%2E+2026a%2E+Concept%2Dgrounded+detection+of+vaccine+misinformation+in+multi)
22. Surendrabikram Thapa, Kritesh Rauniyar, Hariram Veeramani, Aditya Shah, Imran Razzak, and Usman Naseem. 2024. Did you tell a deadly lie? evaluating large language models for health misinformation identification. In International Conference on Web Information Systems Engineering, pages 391–405. Sprin… [find](https://scholar.google.com/scholar?q=Surendrabikram+Thapa%2C+Kritesh+Rauniyar%2C+Hariram+Veeramani%2C+Aditya+Shah%2C+Imran+Razzak%2C+and+Usman+Naseem%2E+2024%2E+Did+you+tell+a+deadly+lie%3F+evaluating+large+language+models+for+health+misinformation+iden)
23. Surendrabikram Thapa, Shuvam Shiwakoti, Siddhant Bikram Shah, Surabhi Adhikari, Hariram Veeramani, Mehwish Nasim, and Usman Naseem. 2025. Large language models (llm) in computational social science: prospects, current state, and challenges. Social Network Analysis and Mining, 15(1):1–30. [find](https://scholar.google.com/scholar?q=Surendrabikram+Thapa%2C+Shuvam+Shiwakoti%2C+Siddhant+Bikram+Shah%2C+Surabhi+Adhikari%2C+Hariram+Veeramani%2C+Mehwish+Nasim%2C+and+Usman+Naseem%2E+2025%2E+Large+language+models+%28llm%29+in+computational+social+science%3A+p)
24. Surendrabikram Thapa, Shuvam Shiwakoti, Siddhant Bikram Shah, Kritesh Rauniyar, Laxmi Thapa, Surabhi Adhikari, Kristina T. Johnson, Ali Hür-riyetoğlu, Hristo Tanev, and Usman Naseem. 2026b. [Multimodal identification of vaccine content stance on social media](https://md.research.elamur.ai/free/en/papers/thapa-2026-multimodal-identification-vaccine-content-stance/). In Proceedings of the 9th Workshop on Ev…
25. Zuhui Wang, Zhaozheng Yin, and Young Anna Argyris. 2020. Detecting medical misinformation on social media using multimodal deep learning. IEEE journal of biomedical and health informatics, 25(6):2193– 2203. [find](https://scholar.google.com/scholar?q=Zuhui+Wang%2C+Zhaozheng+Yin%2C+and+Young+Anna+Argyris%2E+2020%2E+Detecting+medical+misinformation+on+social+media+using+multimodal+deep+learning%2E+IEEE+journal+of+biomedical+and+health+informatics%2C+25%286%29%3A2193%E2%80%93)
26. Haofei Yu, Zhengyang Qi, Lawrence Keunho Jang, Russ Salakhutdinov, Louis-Philippe Morency, and Paul Pu Liang. 2024. Mmoe: Enhancing multimodal models with mixtures of multimodal interaction experts. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1000… [find](https://scholar.google.com/scholar?q=Haofei+Yu%2C+Zhengyang+Qi%2C+Lawrence+Keunho+Jang%2C+Russ+Salakhutdinov%2C+Louis%2DPhilippe+Morency%2C+and+Paul+Pu+Liang%2E+2024%2E+Mmoe%3A+Enhancing+multimodal+models+with+mixtures+of+multimodal+interaction+experts%2E+I)
27. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. [Sigmoid loss for language image pre-training](https://md.research.elamur.ai/free/en/papers/zhai-2023-sigmoid-loss-language-image-pre/). In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986.
28. Li Zhang, Haimeng Fan, Chengxia Peng, Guozheng Rao, and Qing Cong. 2020. Sentiment analysis methods for hpv vaccines related tweets based on transfer learning. In Healthcare, volume 8, page 307. MDPI. *Table 4: The specific prompt templates designed for the multi-agent refinement mechanism.* Agent R… [find](https://scholar.google.com/scholar?q=Li+Zhang%2C+Haimeng+Fan%2C+Chengxia+Peng%2C+Guozheng+Rao%2C+and+Qing+Cong%2E+2020%2E+Sentiment+analysis+methods+for+hpv+vaccines+related+tweets+based+on+transfer+learning%2E+In+Healthcare%2C+volume+8%2C+page+307%2E+MDPI%2E)

*Updated 1 November 2026*