---
title: "Domain Adaptation of Image Encoder for Multimodal Manga"
description: "This student-workshop paper proposes adapting a vision encoder to the manga domain by borrowing four self-supervised… 24 references."
authors: ["Kota Manabe", "Tomoyuki Kajiwara", "Takashi Ninomiya", "Isao Goto", "Shonosuke Ishiwatari", "Hiroshi Noji"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)"
source: "https://aclanthology.org/2026.eacl-srw.3/"
doi: "10.18653/v1/2026.eacl-srw.3"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/manabe-2026-domain-adaptation-image-encoder-multimodal/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/manabe-2026-domain-adaptation-image-encoder-multimodal/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/28/) / Domain Adaptation of Image Encoder for Multimodal Manga Translation

# Domain Adaptation of Image Encoder for Multimodal Manga Translation

**Authors:** [Kota Manabe](https://md.research.elamur.ai/free/en/authors/kota-manabe/), [Tomoyuki Kajiwara](https://md.research.elamur.ai/free/en/authors/tomoyuki-kajiwara/), [Takashi Ninomiya](https://md.research.elamur.ai/free/en/authors/takashi-ninomiya/), [Isao Goto](https://md.research.elamur.ai/free/en/authors/isao-goto/), Shonosuke Ishiwatari and [Hiroshi Noji](https://md.research.elamur.ai/free/en/authors/hiroshi-noji/) · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-srw.3/) · [PDF](https://aclanthology.org/2026.eacl-srw.3.pdf) · [DOI 10.18653/v1/2026.eacl-srw.3](https://doi.org/10.18653/v1/2026.eacl-srw.3) · [OpenAlex](https://openalex.org/W7140128165)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Manga translation systems often struggle because speech-balloon text alone omits subjects and context. Existing multimodal approaches either convert images into tags or captions, or rely on general-purpose image encoders that do not grasp how manga pages are organized. This student-workshop paper proposes adapting a vision encoder to the manga domain by borrowing four self-supervised objectives from NLP and recasting them at the panel level. Panel matching CL (PmCL) comes from contrastive learning, Masked Panel Modeling (MPM) from MLM, Next Panel Prediction (NPP) from NSP, and Permutation Panel Modeling (PPM) from PLM. The resulting encoders (ViT and CLIP variants) supply visual features to an mT5-based Japanese→English multimodal translation model. The authors test serial and parallel ways of fusing these features with text, using pages from the OpenMantra dataset. Domain-adapted encoders generally raise BLEU and xCOMET over both a text-only model and a model with an unadapted encoder. Combining all four objectives works best. This suggests that panel structure and reading order are useful signals for translation.

## Key points
- The four objectives operate on panels rather than tokens. For example, PmCL masks panels on a page and pulls the masked region's embedding toward the true panel while pushing it away from other panels.
- With mT5-base, adapted encoders beat the general-purpose encoder in 12 of 16 configurations; with mT5-large, they did so in only 5. The gains were larger for the smaller model.
- The "all" setting, which combines PmCL, MPM, NPP and PPM, gave the best average scores.
- Serial (early) fusion of visual features consistently outperformed parallel fusion, by more than 1 BLEU point on average. ViT was slightly ahead of CLIP on BLEU.
- In a qualitative example (mT5-large, vit-all-serial), the adapted model correctly rendered a character's gender as "he" where the baseline did not. Stated limitations include reliance on automatic metrics only and no tests yet with MLLMs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The University of Osaka](https://md.research.elamur.ai/free/en/universities/the-university-of-osaka/) — University, Osaka, Japan · 15 works · 517 citations
- [Ehime University](https://md.research.elamur.ai/free/en/universities/ehime-university/) — University, Matsuyama, Japan · 5 works · 3 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works

## Cite

Cite as: Kota Manabe, Tomoyuki Kajiwara, Takashi Ninomiya, Isao Goto, Shonosuke Ishiwatari and Hiroshi Noji. 2026. Domain Adaptation of Image Encoder for Multimodal Manga Translation. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{manabe2026domain,
  title = {{Domain Adaptation of Image Encoder for Multimodal Manga Translation}},
  author = {Kota Manabe, Tomoyuki Kajiwara, Takashi Ninomiya, Isao Goto, Shonosuke Ishiwatari and Hiroshi Noji},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-srw.3},
  url = {https://doi.org/10.18653/v1/2026.eacl-srw.3}
}
```

- [← previous Does Translation Preserve Sentiment? An Analysis of Arabic-English Cross-Lingual Classification](https://md.research.elamur.ai/free/en/papers/mubarak-2026-does-translation-preserve-sentiment-analysis/)
- [next → Domain-Aware Error Correction for Citation NER in Medieval Hebrew Responsa](https://md.research.elamur.ai/free/en/papers/liebeskind-2026-domain-aware-error-correction-citation/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/28/)

## References

24 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Po-Yao Huang, Frederick Liu, Sz-Rung Shiang, Jean Oh, and Chris Dyer. 2016. [Attention-based Multimodal Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/huang-2016-attention-based-multimodal-neural-machine/). In Proceedings of the First Conference on Machine Translation, pages 639–645.
2. Iacer Calixto and Qun Liu. 2017. Incorporating Global Visual Features into Attention-based Neural Machine Translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 992–1003. [find](https://scholar.google.com/scholar?q=Iacer+Calixto+and+Qun+Liu%2E+2017%2E+Incorporating+Global+Visual+Features+into+Attention%2Dbased+Neural+Machine+Translation%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language+Pr)
3. Hiroto Kaino, Soichiro Sugihara, Tomoyuki Kajiwara, Takashi Ninomiya, Joshua B. Tanner, and Shonosuke Ishiwatari. 2024. Utilizing Longer Context than Speech Bubbles in Automated Manga Translation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resour… [find](https://scholar.google.com/scholar?q=Hiroto+Kaino%2C+Soichiro+Sugihara%2C+Tomoyuki+Kajiwara%2C+Takashi+Ninomiya%2C+Joshua+B%2E+Tanner%2C+and+Shonosuke+Ishiwatari%2E+2024%2E+Utilizing+Longer+Context+than+Speech+Bubbles+in+Automated+Manga+Translation%2E+In+)
4. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. [A simple framework for contrastive learning of visual representations](https://md.research.elamur.ai/free/en/papers/chen-2020-simple-framework-contrastive-learning-visual/). In Proceedings of the 37th International Conference on Machine Learning. JMLR.org.
5. Hui Li and Xiao-Jun Wu. 2019. DenseFuse: A Fusion Approach to Infrared and Visible Images. IEEE Transactions on Image Processing, 28(5):2614–2623. [find](https://scholar.google.com/scholar?q=Hui+Li+and+Xiao%2DJun+Wu%2E+2019%2E+DenseFuse%3A+A+Fusion+Approach+to+Infrared+and+Visible+Images%2E+IEEE+Transactions+on+Image+Processing%2C+28%285%29%3A2614%E2%80%932623%2E)
6. Xuxin Cheng, Ziyu Yao, Yifei Xin, Hao An, Hongxiang Li, Yaowei Li, and Yuexian Zou. 2024. Soul-Mix: Enhancing Multimodal Machine Translation with Manifold Mixup. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 11283–11294. [find](https://scholar.google.com/scholar?q=Xuxin+Cheng%2C+Ziyu+Yao%2C+Yifei+Xin%2C+Hao+An%2C+Hongxiang+Li%2C+Yaowei+Li%2C+and+Yuexian+Zou%2E+2024%2E+Soul%2DMix%3A+Enhancing+Multimodal+Machine+Translation+with+Manifold+Mixup%2E+In+Proceedings+of+the+62nd+Annual+Meet)
7. Philip Lippmann, Konrad Skublicki, Joshua Tanner, Shonosuke Ishiwatari, and Jie Yang. 2025. Context-Informed Machine Translation of Manga using Multimodal Large Language Models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 3444–3464. [find](https://scholar.google.com/scholar?q=Philip+Lippmann%2C+Konrad+Skublicki%2C+Joshua+Tanner%2C+Shonosuke+Ishiwatari%2C+and+Jie+Yang%2E+2025%2E+Context%2DInformed+Machine+Translation+of+Manga+using+Multimodal+Large+Language+Models%2E+In+Proceedings+of+the+)
8. Gokulesh Danapal, Giovanni A. Santos, João Paulo C. L. da Costa, Bruno J. G. Praciano, and Gabriel P. M. Pinheiro. 2020. Sensor fusion of camera and LiDAR raw data for vehicle detection. In 2020 Workshop on Communication Networks and Power Systems, pages 1–6. [find](https://scholar.google.com/scholar?q=Gokulesh+Danapal%2C+Giovanni+A%2E+Santos%2C+Jo%C3%A3o+Paulo+C%2E+L%2E+da+Costa%2C+Bruno+J%2E+G%2E+Praciano%2C+and+Gabriel+P%2E+M%2E+Pinheiro%2E+2020%2E+Sensor+fusion+of+camera+and+LiDAR+raw+data+for+vehicle+detection%2E+In+2020+Works)
9. Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, page 807–814. [find](https://scholar.google.com/scholar?q=Vinod+Nair+and+Geoffrey+E%2E+Hinton%2E+2010%2E+Rectified+linear+units+improve+restricted+boltzmann+machines%2E+In+Proceedings+of+the+27th+International+Conference+on+International+Conference+on+Machine+Learni)
10. Nithyasri Narasimhan and Sagarika Singh. 2025. Crossing Language Borders: A Pipeline for Indonesian Manhwa Translation. arXiv:2501.01629. — [arXiv:2501.01629](https://arxiv.org/abs/2501.01629) [find](https://scholar.google.com/scholar?q=Nithyasri+Narasimhan+and+Sagarika+Singh%2E+2025%2E+Crossing+Language+Borders%3A+A+Pipeline+for+Indonesian+Manhwa+Translation%2E+arXiv%3A2501%2E01629%2E+%E2%80%94+%5BarXiv%3A2501%2E01629%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2501%2E01629%29)
11. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). In Advances in Neural Information Processing Systems.
12. Matt Post. 2018. [A Call for Clarity in Reporting BLEU Scores](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/). In Proceedings of the Third Conference on Machine Translation, pages 186–191.
13. Shaowei Yao and Xiaojun Wan. 2020. [Multimodal Transformer for Multimodal Machine Translation](https://md.research.elamur.ai/free/en/papers/yao-2020-multimodal-transformer-multimodal-machine-translation/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4346– 4350.
14. Masaki Saito and Yusuke Matsui. 2015. Illustration2Vec: a semantic vector representation of illustrations. In SIGGRAPH Asia 2015 Technical Briefs. [find](https://scholar.google.com/scholar?q=Masaki+Saito+and+Yusuke+Matsui%2E+2015%2E+Illustration2Vec%3A+a+semantic+vector+representation+of+illustrations%2E+In+SIGGRAPH+Asia+2015+Technical+Briefs%2E)
15. Fabien Scalzo, George Bebis, Mircea Nicolescu, Leandro Loss, and Alireza Tavakkoli. 2008. Feature Fusion Hierarchies for gender classification. In 2008 19th International Conference on Pattern Recognition, pages 1–4. [find](https://scholar.google.com/scholar?q=Fabien+Scalzo%2C+George+Bebis%2C+Mircea+Nicolescu%2C+Leandro+Loss%2C+and+Alireza+Tavakkoli%2E+2008%2E+Feature+Fusion+Hierarchies+for+gender+classification%2E+In+2008+19th+International+Conference+on+Pattern+Recogni)
16. Yongjing Yin, Fandong Meng, Jinsong Su, Chulun Zhou, Zhengyuan Yang, Jie Zhou, and Jiebo Luo. 2020. A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3025–3035. [find](https://scholar.google.com/scholar?q=Yongjing+Yin%2C+Fandong+Meng%2C+Jinsong+Su%2C+Chulun+Zhou%2C+Zhengyuan+Yang%2C+Jie+Zhou%2C+and+Jiebo+Luo%2E+2020%2E+A+Novel+Graph%2Dbased+Multi%2Dmodal+Fusion+Encoder+for+Neural+Machine+Translation%2E+In+Proceedings+of+the)
17. Josef Steinbaeck, Christian Steger, Gerald Holweg, and Norbert Druml. 2018. Design of a Low-Level Radar and Time-of-Flight Sensor Fusion Framework. In 2018 21st Euromicro Conference on Digital System Design, pages 268–275. [find](https://scholar.google.com/scholar?q=Josef+Steinbaeck%2C+Christian+Steger%2C+Gerald+Holweg%2C+and+Norbert+Druml%2E+2018%2E+Design+of+a+Low%2DLevel+Radar+and+Time%2Dof%2DFlight+Sensor+Fusion+Framework%2E+In+2018+21st+Euromicro+Conference+on+Digital+System+)
18. Zhuosheng Zhang, Kehai Chen, Rui Wang, Masao Utiyama, Eiichiro Sumita, Zuchao Li, and Hai Zhao. 2020. Neural Machine Translation with Universal Visual Representation. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhuosheng+Zhang%2C+Kehai+Chen%2C+Rui+Wang%2C+Masao+Utiyama%2C+Eiichiro+Sumita%2C+Zuchao+Li%2C+and+Hai+Zhao%2E+2020%2E+Neural+Machine+Translation+with+Universal+Visual+Representation%2E+In+International+Conference+on+Le)
19. Umut Sulubacak, Ozan Caglayan, Stig-Arne Grönroos, Aku Rouhe, Desmond Elliott, Lucia Specia, and Jörg Tiedemann. 2020. Multimodal Machine Translation through Visuals and Speech. Machine Translation, pages 97–147. [find](https://scholar.google.com/scholar?q=Umut+Sulubacak%2C+Ozan+Caglayan%2C+Stig%2DArne+Gr%C3%B6nroos%2C+Aku+Rouhe%2C+Desmond+Elliott%2C+Lucia+Specia%2C+and+J%C3%B6rg+Tiedemann%2E+2020%2E+Multimodal+Machine+Translation+through+Visuals+and+Speech%2E+Machine+Translation%2C+p)
20. Jörg Tiedemann and Yves Scherrer. 2017. [Neural Machine Translation with Extended Context](https://md.research.elamur.ai/free/en/papers/tiedemann-2017-neural-machine-translation-extended-context/). In Proceedings of the Third Workshop on Discourse in Machine Translation, pages 82–92.
21. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2019. [Representation Learning with Contrastive Predictive Coding](https://md.research.elamur.ai/free/en/papers/oord-2018-representation-learning-contrastive-predictive-coding/). arXiv:1807.03748.
22. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. [Transf…](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
23. Zhiyong Wu, Lingpeng Kong, Wei Bi, Xiang Li, and Ben Kao. 2021. [Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation](https://md.research.elamur.ai/free/en/papers/wu-2021-good-misconceived-reasons-empirical-revisiting/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internation…
24. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. [mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling…

*Updated 1 November 2026*