---
title: "Memory-Guided Hard Data Augmentation for Multimodal Named"
description: "They propose Memory-Guided Hard Data Augmentation, a five-stage pipeline: K-fold cross-validation finds the… 87.13 F1. 26 references."
authors: ["Xinyu Liu", "Kai fu", "Yinghan Shi", "Quanyou Chu", "Ming Du", "Hongya Wang", "Xiaojun Meng", "Jiansheng Wei", "Yanghua Xiao", "Bo Xu"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1075/"
doi: "10.18653/v1/2026.findings-acl.1075"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2026-memory-guided-hard-data-augmentation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2026-memory-guided-hard-data-augmentation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/36/) / Memory-Guided Hard Data Augmentation for Multimodal Named Entity Recognition

# Memory-Guided Hard Data Augmentation for Multimodal Named Entity Recognition

**Authors:** [Xinyu Liu](https://md.research.elamur.ai/free/en/authors/xinyu-liu/), Kai fu, Yinghan Shi, Quanyou Chu, Ming Du, Hongya Wang, [Xiaojun Meng](https://md.research.elamur.ai/free/en/authors/xiaojun-meng/), [Jiansheng Wei](https://md.research.elamur.ai/free/en/authors/jiansheng-wei/), [Yanghua Xiao](https://md.research.elamur.ai/free/en/authors/yanghua-xiao/) and [Bo Xu](https://md.research.elamur.ai/free/en/authors/bo-xu/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1075/) · [PDF](https://aclanthology.org/2026.findings-acl.1075.pdf) · [DOI 10.18653/v1/2026.findings-acl.1075](https://doi.org/10.18653/v1/2026.findings-acl.1075) · [OpenAlex](https://openalex.org/W7166855293)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper targets data augmentation for Multimodal Named Entity Recognition (MNER), where images help disambiguate text. Existing augmentation methods apply the same random perturbations to every model. The authors show that this misses many errors: across HVPNeT, MKGformer and AMNet, roughly 30% of each model's hard examples are unique to that architecture.

They propose Memory-Guided Hard Data Augmentation, a five-stage pipeline:
1. K-fold cross-validation finds the training samples the target model gets wrong.
2. These failures go into a Memory Tree, where an LLM plus clustering groups individual mistakes into broader error patterns.
3. Those patterns drive targeted generation of new text–image samples, using LLMs and diffusion models.
4. The new samples are checked at the label, text and text–image levels.
5. Hard samples whose augmentations fail these checks are sent back for another round of generation.

On Twitter-2015 and Twitter-2017, the framework consistently improves several MNER backbones and beats earlier augmentation methods. The authors frame this as a methodological advance in augmentation rather than an absolute state of the art. Its value lies in fixing each model's own blind spots instead of expanding data generically.

## Key points
- **Model-specific errors:** About 30% of hard instances are unique to each backbone (HVPNeT, MKGformer, AMNet). For HVPNeT, errors shared by all three models make up under 43% of its failures.
- **Error taxonomy:** Rule-based coarse classification sorts errors into four types: entity hallucination, boundary detection failure, entity omission and type confusion. The LLM then abstracts these into finer patterns, such as image–text cue misalignment.
- **Ablation results** (Twitter-2017, HVPNeT backbone): the full framework reaches 87.13 F1. Removing components lowers it as follows:

  | Component removed | F1 |
  |---|---|
  | Data filtering | 85.30 (largest drop, mainly in precision) |
  | Iterative augmentation | 86.42 |
  | Model-aware mining | 86.52 |
  | Memory Tree | 86.83 |

- **Filtering model:** Llama-3-8B is reported as the filter LLM, with 83.35 F1 in a filtering-model comparison.
- **Limitations:** Experiments cover only loosely aligned Twitter data. Hallucinations from the LLM or diffusion model can still introduce noise into the augmented data.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 167 works · 20,987 citations
- [Huawei Technologies](https://md.research.elamur.ai/free/en/companies/huawei-technologies/) — Company, Shenzhen, China · 75 works · 12,975 citations
- [Donghua University](https://md.research.elamur.ai/free/en/universities/donghua-university/) — University, Shanghai, China · 2 works

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works · [🏷️ Named entity recognition](https://md.research.elamur.ai/free/en/topics/named-entity-recognition/) · 200 works

## Cite

Cite as: Xinyu Liu, Kai fu, Yinghan Shi, Quanyou Chu, Ming Du, Hongya Wang, Xiaojun Meng, Jiansheng Wei, Yanghua Xiao and Bo Xu. 2026. Memory-Guided Hard Data Augmentation for Multimodal Named Entity Recognition. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liu2026memory,
  title = {{Memory-Guided Hard Data Augmentation for Multimodal Named Entity Recognition}},
  author = {Xinyu Liu, Kai fu, Yinghan Shi, Quanyou Chu, Ming Du, Hongya Wang, Xiaojun Meng, Jiansheng Wei, Yanghua Xiao and Bo Xu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1075},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1075}
}
```

- [← previous Memory-Efficient Structured Backpropagation for On-Device LLM Fine-Tuning](https://md.research.elamur.ai/free/en/papers/park-2026-memory-efficient-structured-backpropagation-device/)
- [next → MentalMARBERT: Domain-Adaptive Pre-training and Two-Stage Fine-Tuning for Arabic Mental Health Disorders Detection](https://md.research.elamur.ai/free/en/papers/almalki-2026-mentalmarbert-domain-adaptive-pre-training/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/36/)

## References

26 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Xiang Chen, Ningyu Zhang, Lei Li, Shumin Deng, Chuanqi Tan, Changliang Xu, Fei Huang, Luo Si, and Huajun Chen. 2022a. Hybrid transformer with multi-level fusion for multimodal knowledge graph completion. In Proceedings of the 45th international ACM SIGIR conference on research and development in inf… [find](https://scholar.google.com/scholar?q=Xiang+Chen%2C+Ningyu+Zhang%2C+Lei+Li%2C+Shumin+Deng%2C+Chuanqi+Tan%2C+Changliang+Xu%2C+Fei+Huang%2C+Luo+Si%2C+and+Huajun+Chen%2E+2022a%2E+Hybrid+transformer+with+multi%2Dlevel+fusion+for+multimodal+knowledge+graph+completi)
2. Xiang Chen, Ningyu Zhang, Lei Li, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022b. Good visual guidance makes a better extractor: Hierarchical visual prefix for multimodal entity and relation extraction. arXiv preprint arXiv:2205.03521. — [arXiv:2205.03521](https://arxiv.org/abs/2205.03521) [find](https://scholar.google.com/scholar?q=Xiang+Chen%2C+Ningyu+Zhang%2C+Lei+Li%2C+Yunzhi+Yao%2C+Shumin+Deng%2C+Chuanqi+Tan%2C+Fei+Huang%2C+Luo+Si%2C+and+Huajun+Chen%2E+2022b%2E+Good+visual+guidance+makes+a+better+extractor%3A+Hierarchical+visual+prefix+for+multimo)
3. Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. 2019. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 113–123. [find](https://scholar.google.com/scholar?q=Ekin+D+Cubuk%2C+Barret+Zoph%2C+Dandelion+Mane%2C+Vijay+Vasudevan%2C+and+Quoc+V+Le%2E+2019%2E+Autoaugment%3A+Learning+augmentation+strategies+from+data%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+vision+a)
4. Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. 2020. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702–703. [find](https://scholar.google.com/scholar?q=Ekin+D+Cubuk%2C+Barret+Zoph%2C+Jonathon+Shlens%2C+and+Quoc+V+Le%2E+2020%2E+Randaugment%3A+Practical+automated+data+augmentation+with+a+reduced+search+space%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+v)
5. Yinglong Dai, Feng Gao, and Daojian Zeng. 2023. An alignment and matching network with hierarchical visual features for multimodal named entity and relation extraction. In International Conference on Neural Information Processing, pages 298–310. Springer. [find](https://scholar.google.com/scholar?q=Yinglong+Dai%2C+Feng+Gao%2C+and+Daojian+Zeng%2E+2023%2E+An+alignment+and+matching+network+with+hierarchical+visual+features+for+multimodal+named+entity+and+relation+extraction%2E+In+International+Conference+on+)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies,…
7. Bosheng Ding, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu, Anh Tuan Luu, and Shafiq Joty. 2024. Data augmentation using large language models: Data perspectives, learning paradigms and challenges. arXiv preprint arXiv:2403.02990. [find](https://scholar.google.com/scholar?q=Bosheng+Ding%2C+Chengwei+Qin%2C+Ruochen+Zhao%2C+Tianze+Luo%2C+Xinze+Li%2C+Guizhen+Chen%2C+Wenhan+Xia%2C+Junjie+Hu%2C+Anh+Tuan+Luu%2C+and+Shafiq+Joty%2E+2024%2E+Data+augmentation+using+large+language+models%3A+Data+perspectiv)
8. Xiaoshuai Hao, Yi Zhu, Srikar Appalaraju, Aston Zhang, Wanqian Zhang, Bo Li, and Mu Li. 2023. Mixgen: A new multi-modal data augmentation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 379–389. [find](https://scholar.google.com/scholar?q=Xiaoshuai+Hao%2C+Yi+Zhu%2C+Srikar+Appalaraju%2C+Aston+Zhang%2C+Wanqian+Zhang%2C+Bo+Li%2C+and+Mu+Li%2E+2023%2E+Mixgen%3A+A+new+multi%2Dmodal+data+augmentation%2E+In+Proceedings+of+the+IEEE%2FCVF+winter+conference+on+applicati)
9. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. [Bidirectional lstm-crf models for sequence tagging](https://md.research.elamur.ai/free/en/papers/huang-2015-bidirectional-lstm-crf-models-sequence/). arxiv 2015. arXiv preprint arXiv:1508.01991.
10. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. [Neural architectures for named entity recognition](https://md.research.elamur.ai/free/en/papers/lample-2016-neural-architectures-named-entity-recognition/). arXiv preprint arXiv:1603.01360.
11. Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2020. A survey on deep learning for named entity recognition. IEEE transactions on knowledge and data engineering, 34(1):50–70. [find](https://scholar.google.com/scholar?q=Jing+Li%2C+Aixin+Sun%2C+Jianglei+Han%2C+and+Chenliang+Li%2E+2020%2E+A+survey+on+deep+learning+for+named+entity+recognition%2E+IEEE+transactions+on+knowledge+and+data+engineering%2C+34%281%29%3A50%E2%80%9370%2E)
12. Ziyan Li, Jianfei Yu, Jia Yang, Wenya Wang, Li Yang, and Rui Xia. 2024. Generative multimodal data augmentation for low-resource multimodal named entity recognition. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 7336– 7345. [find](https://scholar.google.com/scholar?q=Ziyan+Li%2C+Jianfei+Yu%2C+Jia+Yang%2C+Wenya+Wang%2C+Li+Yang%2C+and+Rui+Xia%2E+2024%2E+Generative+multimodal+data+augmentation+for+low%2Dresource+multimodal+named+entity+recognition%2E+In+Proceedings+of+the+32nd+ACM+Int)
13. Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. 2018. [Visual attention model for name tagging in multimodal social media](https://md.research.elamur.ai/free/en/papers/lu-2018-visual-attention-model-name-tagging/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1990–1999.
14. Xuezhe Ma and Eduard Hovy. 2016. [End-to-end sequence labeling via bi-directional lstm-cnns-crf](https://md.research.elamur.ai/free/en/papers/ma-2016-end-end-sequence-labeling-bi/). arXiv preprint arXiv:1603.01354.
15. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695. [find](https://scholar.google.com/scholar?q=Robin+Rombach%2C+Andreas+Blattmann%2C+Dominik+Lorenz%2C+Patrick+Esser%2C+and+Bj%C3%B6rn+Ommer%2E+2022%2E+High%2Dresolution+image+synthesis+with+latent+diffusion+models%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+compu)
16. Lin Sun, Jiquan Wang, Kai Zhang, Yindu Su, and Fangsheng Weng. 2021. Rpbert: a text-image relation propagation-based bert model for multimodal ner. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 13860–13868. [find](https://scholar.google.com/scholar?q=Lin+Sun%2C+Jiquan+Wang%2C+Kai+Zhang%2C+Yindu+Su%2C+and+Fangsheng+Weng%2E+2021%2E+Rpbert%3A+a+text%2Dimage+relation+propagation%2Dbased+bert+model+for+multimodal+ner%2E+In+Proceedings+of+the+AAAI+conference+on+artificial+)
17. Xinyu Wang, Min Gui, Yong Jiang, Zixia Jia, Nguyen Bach, Tao Wang, Zhongqiang Huang, and Kewei Tu. 2022. Ita: Image-text alignments for multi-modal named entity recognition. In Proceedings of the 2022 conference of the North American chapter of the association for computational linguistics: human la… [find](https://scholar.google.com/scholar?q=Xinyu+Wang%2C+Min+Gui%2C+Yong+Jiang%2C+Zixia+Jia%2C+Nguyen+Bach%2C+Tao+Wang%2C+Zhongqiang+Huang%2C+and+Kewei+Tu%2E+2022%2E+Ita%3A+Image%2Dtext+alignments+for+multi%2Dmodal+named+entity+recognition%2E+In+Proceedings+of+the+2022)
18. Zaitian Wang, Pengfei Wang, Kunpeng Liu, Pengyang Wang, Yanjie Fu, Chang-Tien Lu, Charu C Aggarwal, Jian Pei, and Yuanchun Zhou. 2025. A comprehensive survey on data augmentation. IEEE Transactions on Knowledge and Data Engineering. [find](https://scholar.google.com/scholar?q=Zaitian+Wang%2C+Pengfei+Wang%2C+Kunpeng+Liu%2C+Pengyang+Wang%2C+Yanjie+Fu%2C+Chang%2DTien+Lu%2C+Charu+C+Aggarwal%2C+Jian+Pei%2C+and+Yuanchun+Zhou%2E+2025%2E+A+comprehensive+survey+on+data+augmentation%2E+IEEE+Transactions+on)
19. Bo Xu, Shizhou Huang, Ming Du, Hongya Wang, Hui Song, Yanghua Xiao, and Xin Lin. 2023. A unified visual prompt tuning framework with mixture-of-experts for multimodal information extraction. In International Conference on Database Systems for Advanced Applications, pages 544–554. Springer. [find](https://scholar.google.com/scholar?q=Bo+Xu%2C+Shizhou+Huang%2C+Ming+Du%2C+Hongya+Wang%2C+Hui+Song%2C+Yanghua+Xiao%2C+and+Xin+Lin%2E+2023%2E+A+unified+visual+prompt+tuning+framework+with+mixture%2Dof%2Dexperts+for+multimodal+information+extraction%2E+In+Intern)
20. Bo Xu, Shizhou Huang, Chaofeng Sha, and Hongya Wang. 2022. Maf: A general matching and alignment framework for multimodal named entity recognition. In Proceedings of the fifteenth ACM international conference on web search and data mining, pages 1215–1223. [find](https://scholar.google.com/scholar?q=Bo+Xu%2C+Shizhou+Huang%2C+Chaofeng+Sha%2C+and+Hongya+Wang%2E+2022%2E+Maf%3A+A+general+matching+and+alignment+framework+for+multimodal+named+entity+recognition%2E+In+Proceedings+of+the+fifteenth+ACM+international+co)
21. Bo Xu, Haiqi Jiang, Jie Wei, Hongyu Jing, Ming Du, Hui Song, Hongya Wang, and Yanghua Xiao. 2025. Enhancing multimodal named entity recognition through adaptive mixup image augmentation. In Proceedings of the 31st International Conference on Computational Linguistics, pages 1802–1812. [find](https://scholar.google.com/scholar?q=Bo+Xu%2C+Haiqi+Jiang%2C+Jie+Wei%2C+Hongyu+Jing%2C+Ming+Du%2C+Hui+Song%2C+Hongya+Wang%2C+and+Yanghua+Xiao%2E+2025%2E+Enhancing+multimodal+named+entity+recognition+through+adaptive+mixup+image+augmentation%2E+In+Proceeding)
22. Jianfei Yu, Jing Jiang, Li Yang, and Rui Xia. 2020. [Improving multimodal named entity recognition via entity span detection with unified multimodal transformer](https://md.research.elamur.ai/free/en/papers/yu-2020-improving-multimodal-named-entity-recognition/). Association for Computational Linguistics.
23. Dong Zhang, Suzhong Wei, Shoushan Li, Hanqian Wu, Qiaoming Zhu, and Guodong Zhou. 2021. Multimodal graph fusion for named entity recognition with targeted visual guidance. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 14347–14355. [find](https://scholar.google.com/scholar?q=Dong+Zhang%2C+Suzhong+Wei%2C+Shoushan+Li%2C+Hanqian+Wu%2C+Qiaoming+Zhu%2C+and+Guodong+Zhou%2E+2021%2E+Multimodal+graph+fusion+for+named+entity+recognition+with+targeted+visual+guidance%2E+In+Proceedings+of+the+AAAI+c)
24. Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2017. [mixup: Beyond empirical risk minimization](https://md.research.elamur.ai/free/en/papers/zhang-2017-mixup-empirical-risk-minimization/). arXiv preprint arXiv:1710.09412.
25. Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018. Adaptive co-attention network for named entity recognition in tweets. In Proceedings of the AAAI conference on artificial intelligence, volume 32. [find](https://scholar.google.com/scholar?q=Qi+Zhang%2C+Jinlan+Fu%2C+Xiaoyu+Liu%2C+and+Xuanjing+Huang%2E+2018%2E+Adaptive+co%2Dattention+network+for+named+entity+recognition+in+tweets%2E+In+Proceedings+of+the+AAAI+conference+on+artificial+intelligence%2C+volum)
26. Xiaoying Zhou, Yijia Zhang, Zhuang Wang, Mingyu Lu, and Xiaoxia Liu. 2024. Mafn: multi-level attention fusion network for multimodal named entity recognition. Multimedia Tools and Applications, 83(15):45047–45058. [find](https://scholar.google.com/scholar?q=Xiaoying+Zhou%2C+Yijia+Zhang%2C+Zhuang+Wang%2C+Mingyu+Lu%2C+and+Xiaoxia+Liu%2E+2024%2E+Mafn%3A+multi%2Dlevel+attention+fusion+network+for+multimodal+named+entity+recognition%2E+Multimedia+Tools+and+Applications%2C+83%2815%29)

*Updated 1 November 2026*