---
title: "TriFine: A Large-Scale Dataset of — Guan 2025"
description: "Video-guided machine translation (VMT) usually feeds a model dense visual features from many frames."
authors: ["Boyu Guan", "Yining Zhang", "Yang Zhao", "Chengqing Zong"]
year: 2025
venue: "Proceedings of the 31st International Conference on Computational Linguistics"
source: "https://aclanthology.org/2025.coling-main.547/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/guan-2025-trifine-large-scale-dataset-vision/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/guan-2025-trifine-large-scale-dataset-vision/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2025](https://md.research.elamur.ai/free/papers/2025/page/2/) / TriFine: A Large-Scale Dataset of Vision-Audio-Subtitle for Tri-Modal Machine Translation and Benchmark with Fine-Grained Annotated Tags

# TriFine: A Large-Scale Dataset of Vision-Audio-Subtitle for Tri-Modal Machine Translation and Benchmark with Fine-Grained Annotated Tags

**Authors:** Boyu Guan, Yining Zhang, Yang Zhao, [Chengqing Zong](https://md.research.elamur.ai/free/authors/chengqing-zong/) · **Year:** 2025 · **Venue:** Proceedings of the 31st International Conference on Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.coling-main.547/) · [PDF](https://aclanthology.org/2025.coling-main.547.pdf)

**Citations:** 9 in our library, of the 140673 papers whose bibliographies we read

## Summary
Video-guided machine translation (VMT) usually feeds a model dense visual features from many frames. This approach is slow, carries a lot of irrelevant signal, and ignores the soundtrack. The authors argue that compact, targeted cues from both vision and audio work better. To test this, they release TriFine, an English–Chinese VMT corpus of about 1.2M En→Zh and 1.18M Zh→En training entries. Each entry combines a subtitle pair and a 10-second clip with seven automatically produced tags:
- video caption
- location
- action
- entities
- facial expression
- audio sentiment
- audio stress

They chose these tag types through a human study of which cues help disambiguation. They also propose FIAT, a model-agnostic method that adds the tags to the input and uses a soft attention mask, leaving the underlying NMT model unchanged.

FIAT beats text-only translation and coarse frame-encoder baselines on BLEU, METEOR and COMET, and trains at a fraction of their compute cost. The work shows that small, interpretable multimodal signals, including audio, can help translation more than raw video features.

## Key points
- **How tag types were chosen:** three annotators reviewed 500 samples that are hard for text-only MT. Captions were judged helpful in 86.6% of cases and audio pattern in only 11.2%, so pattern was dropped and seven tag types were kept.
- **Seven test sets:** general, ambiguity, and five information-specific sets (audio sentiment, expression, action, location, entities). Data and code are at github.com/BoyuGuan/TriFine.
- **Ambiguity test set results:** FIAT with all tags reaches BLEU 31.24, METEOR 44.89 and COMET 75.93. The text-only baseline scores 29.85, 42.22 and 74.39, and the strongest frame-based encoder (TVE) scores 30.37, 42.73 and 74.45.
- **Training cost:** frame-based encoders (TVE, CVE) need roughly 180–195 GPU hours, while FIAT variants need roughly 9–29.
- **Limitations:** expression tags hurt FIAT compared with the text-only and TVE baselines. FIAT also requires the tags to be supplied at inference, and some noisy data remains in the corpus.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{guan2025trifine,
  title = {{TriFine: A Large-Scale Dataset of Vision-Audio-Subtitle for Tri-Modal Machine Translation and Benchmark with Fine-Grained Annotated Tags}},
  author = {Guan, Boyu and Zhang, Yining and Zhao, Yang and Zong, Chengqing},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.547/}
}
```

- [← previous Transforming Causal LLM into MLM Encoder for Detecting Social Media Manipulation in Telegram](https://md.research.elamur.ai/free/papers/bazdyrev-2025-transforming-causal-llm-into-mlm/)
- [next → TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs](https://md.research.elamur.ai/free/papers/ar-2025-turblimp-turkish-benchmark-linguistic-minimal/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/papers/2025/page/2/)

## References

53 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ozan Caglayan, Menekse Kuyu, Mustafa Sercan Amac, Pranava Madhyastha, Erkut Erdem, Aykut Erdem, and Lucia Specia. 2021. Cross-lingual visual pre-training for multimodal machine translation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1317–1324, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ozan+Caglayan%2C+Menekse+Kuyu%2C+Mustafa+Sercan+Amac%2C+Pranava+Madhyastha%2C+Erkut+Erdem%2C+Aykut+Erdem%2C+and+Lucia+Specia%2E+2021%2E+Cross%2Dlingual+visual+pre%2Dtraining+for+multimodal+machine+translation%2E+In+Proceed)
2. The data was collected in strict adherence to the source’s terms of use and copyright policies, utiliz- Ozan Caglayan, Pranava Madhyastha, Lucia Specia, and Loïc Barrault. 2019. Probing the need for visual ing tools commonly employed in previous research. context in multimodal machine translation. In Pro-Videos were divided into 10-second segments, with ceedings of the 2019 Conference of the North American Chapter of the Association for Computational the number of clips per video limited to avoid the Linguistics: Human Language Technologies, Volume potential substitution of the original video. Users 1 (Long and Short Papers), pages 4159–4170, Min-of our dataset will be required to sign a usage agree-neapolis, Minnesota. Association for Computational ment, stipulating that the dataset is to be used ex-Linguistics. clusively for academic research purposes. Before the dataset is released, we will perform an addi- Dipankar Das, Naveen Mellempudi, Dheevatsa Mudigere, Dhiraj Kalamkar, Sasikanth Avancha, Kunal tional filtering process to eliminate any privacy and Banerjee, Srinivas Sridharan, Karthik Vaidyanathan, copyright-sensitive content to the best extent possi-Bharat Kaul, Evangelos Georganas, et al. 2018. ble. Mixed precision training of convolutional neural net-8223 works using integer operations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=The+data+was+collected+in+strict+adherence+to+the+source%E2%80%99s+terms+of+use+and+copyright+policies%2C+utiliz%2D+Ozan+Caglayan%2C+Pranava+Madhyastha%2C+Lucia+Specia%2C+and+Lo%C3%AFc+Barrault%2E+2019%2E+Probing+the+need+for+v)
3. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648, Atlanta, Georgia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chris+Dyer%2C+Victor+Chahuneau%2C+and+Noah+A%2E+Smith%2E+2013%2E+A+simple%2C+fast%2C+and+effective+reparameterization+of+IBM+model+2%2E+In+Proceedings+of+the+2013+Conference+of+the+North+American+Chapter+of+the+Assoc)
4. Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016. Multi30K: Multilingual English-German image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pages 70– 74, Berlin, Germany. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Desmond+Elliott%2C+Stella+Frank%2C+Khalil+Sima%E2%80%99an%2C+and+Lucia+Specia%2E+2016%2E+Multi30K%3A+Multilingual+English%2DGerman+image+descriptions%2E+In+Proceedings+of+the+5th+Workshop+on+Vision+and+Language%2C+pages+70%E2%80%93+74) [our card](https://md.research.elamur.ai/free/papers/elliott-2016-multi30k-multilingual-english-german-image/)
5. Qingkai Fang and Yang Feng. 2022. Neural machine translation with phrase-level universal visual representations. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5687–5698, Dublin, Ireland. Association for Computational Linguistics. [↗](https://doi.org/10.48448/99wn-5n79)
6. Hao Fei, Qian Liu, Meishan Zhang, Min Zhang, and Tat-Seng Chua. 2023. Scene graph as pivoting: Inference-time image-free unsupervised multimodal machine translation with visual scene hallucination. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5980–5994, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hao+Fei%2C+Qian+Liu%2C+Meishan+Zhang%2C+Min+Zhang%2C+and+Tat%2DSeng+Chua%2E+2023%2E+Scene+graph+as+pivoting%3A+Inference%2Dtime+image%2Dfree+unsupervised+multimodal+machine+translation+with+visual+scene+hallucination%2E+In)
7. Matthieu Futeral, Cordelia Schmid, Ivan Laptev, Benoît Sagot, and Rachel Bawden. 2023. Tackling ambiguity with images: Improved multimodal machine translation and contrastive evaluation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5394–5413, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Matthieu+Futeral%2C+Cordelia+Schmid%2C+Ivan+Laptev%2C+Beno%C3%AEt+Sagot%2C+and+Rachel+Bawden%2E+2023%2E+Tackling+ambiguity+with+images%3A+Improved+multimodal+machine+translation+and+contrastive+evaluation%2E+In+Proceeding)
8. Weiqi Gu, Haiyue Song, Chenhui Chu, and Sadao Kurohashi. 2021. Video-guided machine translation with spatial hierarchical attention network. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop, pages 87–92, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Weiqi+Gu%2C+Haiyue+Song%2C+Chenhui+Chu%2C+and+Sadao+Kurohashi%2E+2021%2E+Video%2Dguided+machine+translation+with+spatial+hierarchical+attention+network%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Associatio)
9. Shoutao Guo, Shaolei Zhang, and Yang Feng. 2024. Decoder-only streaming transformer for simultaneous translation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8851–8864, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shoutao+Guo%2C+Shaolei+Zhang%2C+and+Yang+Feng%2E+2024%2E+Decoder%2Donly+streaming+transformer+for+simultaneous+translation%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Association+for+Computational+Linguis)
10. Tosho Hirasawa, Zhishen Yang, Mamoru Komachi, and Naoaki Okazaki. 2020. Keyframe segmentation and positional encoding for video-guided machine translation challenge 2020. arXiv preprint arXiv:2006.12799. [↗](https://arxiv.org/abs/2006.12799)
11. Xin Huang, Jiajun Zhang, and Chengqing Zong. 2021. Entity-level cross-modal learning improves multimodal machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1067–1080, Punta Cana, Dominican Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xin+Huang%2C+Jiajun+Zhang%2C+and+Chengqing+Zong%2E+2021%2E+Entity%2Dlevel+cross%2Dmodal+learning+improves+multimodal+machine+translation%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2021%2C+)
12. Xin Huang, Jiajun Zhang, and Chengqing Zong. 2023. Contrastive adversarial training for multi-modal machine translation. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 22(6). [↗](https://doi.org/10.1145/3587267)
13. Julia Ive, Pranava Madhyastha, and Lucia Specia. 2019. Distilling translations with visual awareness. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6525– 6538, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1906.07701)
14. Liyan Kang, Luyang Huang, Ningxin Peng, Peihao Zhu, Zewei Sun, Shanbo Cheng, Mingxuan Wang, Degen Huang, and Jinsong Su. 2023. BigVideo: A large-scale video subtitle translation dataset for multimodal machine translation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8456–8473, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Liyan+Kang%2C+Luyang+Huang%2C+Ningxin+Peng%2C+Peihao+Zhu%2C+Zewei+Sun%2C+Shanbo+Cheng%2C+Mingxuan+Wang%2C+Degen+Huang%2C+and+Jinsong+Su%2E+2023%2E+BigVideo%3A+A+large%2Dscale+video+subtitle+translation+dataset+for+multimodal)
15. Mingjie Li, Po-Yao Huang, Xiaojun Chang, Junjie Hu, Yi Yang, and Alex Hauptmann. 2023a. Video pivoting unsupervised multi-modal machine translation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3918–3932. [↗](https://doi.org/10.1109/tpami.2022.3181116)
16. Yi Li, Rameswar Panda, Yoon Kim, Chun-Fu Richard Chen, Rogerio S Feris, David Cox, and Nuno Vasconcelos. 2022a. Valhalla: Visual hallucination for machine translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5216–5226. [find](https://scholar.google.com/scholar?q=Yi+Li%2C+Rameswar+Panda%2C+Yoon+Kim%2C+Chun%2DFu+Richard+Chen%2C+Rogerio+S+Feris%2C+David+Cox%2C+and+Nuno+Vasconcelos%2E+2022a%2E+Valhalla%3A+Visual+hallucination+for+machine+translation%2E+In+Proceedings+of+the+IEEE%2FCVF+C)
17. Yihang Li, Shuichiro Shimizu, Chenhui Chu, Sadao Kurohashi, and Wei Li. 2023b. Video-helpful multimodal machine translation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4281–4299, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yihang+Li%2C+Shuichiro+Shimizu%2C+Chenhui+Chu%2C+Sadao+Kurohashi%2C+and+Wei+Li%2E+2023b%2E+Video%2Dhelpful+multimodal+machine+translation%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+Langu)
18. Yihang Li, Shuichiro Shimizu, Weiqi Gu, Chenhui Chu, and Sadao Kurohashi. 2022b. VISA: An ambiguous subtitles dataset for visual scene-aware machine translation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6735–6743, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Yihang+Li%2C+Shuichiro+Shimizu%2C+Weiqi+Gu%2C+Chenhui+Chu%2C+and+Sadao+Kurohashi%2E+2022b%2E+VISA%3A+An+ambiguous+subtitles+dataset+for+visual+scene%2Daware+machine+translation%2E+In+Proceedings+of+the+Thirteenth+Langu)
19. Yunlong Liang, Fandong Meng, Jinan Xu, Yufeng Chen, and Jie Zhou. 2022. MSCTD: A multimodal sentiment chat translation dataset. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2601–2613, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yunlong+Liang%2C+Fandong+Meng%2C+Jinan+Xu%2C+Yufeng+Chen%2C+and+Jie+Zhou%2E+2022%2E+MSCTD%3A+A+multimodal+sentiment+chat+translation+dataset%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computa)
20. Yupu Liang, Yaping Zhang, Cong Ma, Zhiyang Zhang, Yang Zhao, Lu Xiang, Chengqing Zong, and Yu Zhou. 2024. Document image machine translation with dynamic multi-pre-trained models assembling. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7084–7095, Mexico City, Mexico. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yupu+Liang%2C+Yaping+Zhang%2C+Cong+Ma%2C+Zhiyang+Zhang%2C+Yang+Zhao%2C+Lu+Xiang%2C+Chengqing+Zong%2C+and+Yu+Zhou%2E+2024%2E+Document+image+machine+translation+with+dynamic+multi%2Dpre%2Dtrained+models+assembling%2E+In+Procee)
21. Huan Lin, Fandong Meng, Jinsong Su, Yongjing Yin, Zhengyuan Yang, Yubin Ge, Jie Zhou, and Jiebo Luo. 2020. Dynamic context-guided capsule network for multimodal machine translation. In Proceedings of the 28th ACM International Conference on Multimedia, MM ’20, page 1320–1329, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Huan+Lin%2C+Fandong+Meng%2C+Jinsong+Su%2C+Yongjing+Yin%2C+Zhengyuan+Yang%2C+Yubin+Ge%2C+Jie+Zhou%2C+and+Jiebo+Luo%2E+2020%2E+Dynamic+context%2Dguided+capsule+network+for+multimodal+machine+translation%2E+In+Proceedings+of+)
22. Yuchen Liu, Jiajun Zhang, Hao Xiong, Long Zhou, Zhongjun He, Hua Wu, Haifeng Wang, and Chengqing Zong. 2020. Synchronous speech recognition and speech-to-text translation with interactive decoding. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8417–8424. [find](https://scholar.google.com/scholar?q=Yuchen+Liu%2C+Jiajun+Zhang%2C+Hao+Xiong%2C+Long+Zhou%2C+Zhongjun+He%2C+Hua+Wu%2C+Haifeng+Wang%2C+and+Chengqing+Zong%2E+2020%2E+Synchronous+speech+recognition+and+speech%2Dto%2Dtext+translation+with+interactive+decoding%2E+Pr)
23. Zi Long, Zhenhao Tang, Xianghua Fu, Jian Chen, Shilong Hou, and Jinze Lyu. 2024. Exploring the necessity of visual modality in multimodal machine translation using authentic datasets. arXiv preprint arXiv:2404.06107. [↗](https://arxiv.org/abs/2404.06107)
24. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+7th+International+Conference+on+Learning+Representations%2C+ICLR+2019%2C+New+Orleans%2C+LA%2C+USA%2C+May+6%2D9%2C+2019%2E+OpenReview%2En)
25. Cong Ma, Yaping Zhang, Mei Tu, Yang Zhao, Yu Zhou, and Chengqing Zong. 2023. CCIM: Cross-modal cross-lingual interactive image translation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4959–4965, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Cong+Ma%2C+Yaping+Zhang%2C+Mei+Tu%2C+Yang+Zhao%2C+Yu+Zhou%2C+and+Chengqing+Zong%2E+2023%2E+CCIM%3A+Cross%2Dmodal+cross%2Dlingual+interactive+image+translation%2E+In+Findings+of+the+Association+for+Computational+Linguistics)
26. Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, ShiLiang Zhang, and Xie Chen. 2024. emotion2vec: Self-supervised pre-training for speech emotion representation. In Findings of the Association for Computational Linguistics ACL 2024, pages 15747–15760, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ziyang+Ma%2C+Zhisheng+Zheng%2C+Jiaxin+Ye%2C+Jinchao+Li%2C+Zhifu+Gao%2C+ShiLiang+Zhang%2C+and+Xie+Chen%2E+2024%2E+emotion2vec%3A+Self%2Dsupervised+pre%2Dtraining+for+speech+emotion+representation%2E+In+Findings+of+the+Associa)
27. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. [↗](https://doi.org/10.48550/arxiv.1912.01703) [our card](https://md.research.elamur.ai/free/papers/paszke-2019-pytorch-imperative-style-high-performance/)
28. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Belgium, Brussels. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Matt+Post%2E+2018%2E+A+call+for+clarity+in+reporting+BLEU+scores%2E+In+Proceedings+of+the+Third+Conference+on+Machine+Translation%3A+Research+Papers%2C+pages+186%E2%80%93+191%2C+Belgium%2C+Brussels%2E+Association+for+Computa) [our card](https://md.research.elamur.ai/free/papers/post-2018-call-clarity-reporting-bleu-scores/)
29. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506. [find](https://scholar.google.com/scholar?q=Jeff+Rasley%2C+Samyam+Rajbhandari%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+2020%2E+Deepspeed%3A+System+optimizations+enable+training+deep+learning+models+with+over+100+billion+parameters%2E+In+Proceedings+of+the+26t)
30. Mirco Ravanelli, Titouan Parcollet, Adel Moumen, Sylvain de Langen, Cem Subakan, Peter Plantinga, Yingzhi Wang, Pooneh Mousavi, Luca Della Libera, Artem Ploujnikov, Francesco Paissan, Davide Borra, Salah Zaiem, Zeyu Zhao, Shucong Zhang, Georgios Karakasidis, Sung-Lin Yeh, Pierre Champion, Aku Rouhe, Rudolf Braun, Florian Mai, Juan Zuluaga-Gomez, Seyed Mahed Mousavi, Andreas Nautsch, Xuechen Liu, Sangeet Sagar, Jarod Duret, Salima Mdhaffar, Gaelle Laperriere, Mickael Rouvier, Renato De Mori, and Yannick Esteve. 2024. Open-source conversational ai with speechbrain 1.0. Preprint, arXiv:2407.00463. [↗](https://arxiv.org/abs/2407.00463)
31. Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, Ju-Chieh Chou, Sung-Lin Yeh, Szu-Wei Fu, Chien-Feng Liao, Elena Rastorgueva, François Grondin, William Aris, Hwidong Na, Yan Gao, Renato De Mori, and Yoshua Bengio. 2021. SpeechBrain: A general-purpose speech toolkit. Preprint, arXiv:2106.04624. ArXiv:2106.04624. [↗](https://arxiv.org/abs/2106.04624)
32. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Craig+Stewart%2C+Ana+C+Farinha%2C+and+Alon+Lavie%2E+2020%2E+COMET%3A+A+neural+framework+for+MT+evaluation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+) [our card](https://md.research.elamur.ai/free/papers/rei-2020-comet-neural-framework-mt-evaluation/)
33. Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loïc Barrault, Lucia Specia, and Florian Metze. 2018. How2: a large-scale dataset for multimodal language understanding. In Proceedings of the Workshop on Visually Grounded Interaction and Language (ViGIL). NeurIPS. [↗](https://doi.org/10.48550/arxiv.1811.00347)
34. Huangjun Shen, Liangying Shao, Wenbo Li, Zhibin Lan, Zhanyu Liu, and Jinsong Su. 2024. A survey on multi-modal machine translation: Tasks, methods and challenges. arXiv preprint arXiv:2405.12669. [↗](https://arxiv.org/abs/2405.12669)
35. Ammon Shurtz, Lawry Sorenson, and Stephen D. Richardson. 2024. The effects of pretraining in video-guided machine translation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 15888–15898, Torino, Italia. ELRA and ICCL. [find](https://scholar.google.com/scholar?q=Ammon+Shurtz%2C+Lawry+Sorenson%2C+and+Stephen+D%2E+Richardson%2E+2024%2E+The+effects+of+pretraining+in+video%2Dguided+machine+translation%2E+In+Proceedings+of+the+2024+Joint+International+Conference+on+Computationa)
36. Yanzhi Tian, Xiang Li, Zeming Liu, Yuhang Guo, and Bin Wang. 2023. In-image neural machine translation with segmented pixel sequence-to-sequence model. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15046– 15057, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yanzhi+Tian%2C+Xiang+Li%2C+Zeming+Liu%2C+Yuhang+Guo%2C+and+Bin+Wang%2E+2023%2E+In%2Dimage+neural+machine+translation+with+segmented+pixel+sequence%2Dto%2Dsequence+model%2E+In+Findings+of+the+Association+for+Computational)
37. Jörg Tiedemann, Mikko Aulamo, Daria Bakshandaeva, Michele Boggia, Stig-Arne Grönroos, Tommi Nieminen, Alessandro Raganato Yves Scherrer, Raul Vazquez, and Sami Virpioja. 2023. Democratizing neural machine translation with OPUS-MT. Language Resources and Evaluation, (58):713–755. [↗](https://doi.org/10.1007/s10579-023-09704-w)
38. Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT — Building open translation services for the World. In Proceedings of the 22nd Annual Conferenec of the European Association for Machine Translation (EAMT), Lisbon, Portugal. [find](https://scholar.google.com/scholar?q=J%C3%B6rg+Tiedemann+and+Santhosh+Thottingal%2E+2020%2E+OPUS%2DMT+%E2%80%94+Building+open+translation+services+for+the+World%2E+In+Proceedings+of+the+22nd+Annual+Conferenec+of+the+European+Association+for+Machine+Translati)
39. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+Advances+in+neural+information+processing+)
40. Dexin Wang and Deyi Xiong. 2021. Efficient object-level visual context modeling for multimodal machine translation: Masking irrelevant objects helps grounding. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 2720– 2728. AAAI Press. [↗](https://doi.org/10.48448/06rs-am96)
41. Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. 2019. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4581–4591. [find](https://scholar.google.com/scholar?q=Xin+Wang%2C+Jiawei+Wu%2C+Junkun+Chen%2C+Lei+Li%2C+Yuan%2DFang+Wang%2C+and+William+Yang+Wang%2E+2019%2E+Vatex%3A+A+large%2Dscale%2C+high%2Dquality+multilingual+dataset+for+video%2Dand%2Dlanguage+research%2E+In+Proceedings+of+the+IE)
42. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.emnlp-demos.6) [our card](https://md.research.elamur.ai/free/papers/wolf-2020-transformers-state-art-natural-language/)
43. Zhiyong Wu, Lingpeng Kong, Wei Bi, Xiang Li, and Ben Kao. 2021. Good for misconceived reasons: An empirical revisiting on the need for visual context in multimodal machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6153–6166, Online. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2105.14462)
44. Jian Yang, Hongcheng Guo, Yuwei Yin, Jiaqi Bai, Bing Wang, Jiaheng Liu, Xinnian Liang, LinZheng Chai, Liqun Yang, and Zhoujun Li. 2024. m3P: Towards multimodal multilingual translation with multimodal prompt. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 10858–10871, Torino, Italia. ELRA and ICCL. [find](https://scholar.google.com/scholar?q=Jian+Yang%2C+Hongcheng+Guo%2C+Yuwei+Yin%2C+Jiaqi+Bai%2C+Bing+Wang%2C+Jiaheng+Liu%2C+Xinnian+Liang%2C+LinZheng+Chai%2C+Liqun+Yang%2C+and+Zhoujun+Li%2E+2024%2E+m3P%3A+Towards+multimodal+multilingual+translation+with+multimodal)
45. Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint 2408.01800. [↗](https://arxiv.org/abs/2408.01800) [our card](https://md.research.elamur.ai/free/papers/yao-2024-minicpm-v-gpt-4v/)
46. Donglei Yu, Xiaomian Kang, Yuchen Liu, Yu Zhou, and Chengqing Zong. 2024a. Self-modifying state modeling for simultaneous machine translation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9781–9795, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Donglei+Yu%2C+Xiaomian+Kang%2C+Yuchen+Liu%2C+Yu+Zhou%2C+and+Chengqing+Zong%2E+2024a%2E+Self%2Dmodifying+state+modeling+for+simultaneous+machine+translation%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Associat)
47. Tengfei Yu, Xuebo Liu, Liang Ding, Kehai Chen, Dacheng Tao, and Min Zhang. 2024b. Speech sense disambiguation: Tackling homophone ambiguity in end-to-end speech translation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8020–8035, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tengfei+Yu%2C+Xuebo+Liu%2C+Liang+Ding%2C+Kehai+Chen%2C+Dacheng+Tao%2C+and+Min+Zhang%2E+2024b%2E+Speech+sense+disambiguation%3A+Tackling+homophone+ambiguity+in+end%2Dto%2Dend+speech+translation%2E+In+Proceedings+of+the+62nd)
48. Shaolei Zhang and Yang Feng. 2023. Hidden markov transformer for simultaneous machine translation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. [↗](https://doi.org/10.48550/arxiv.2303.00257)
49. Yuhao Zhang, Chen Xu, Bei Li, Hao Chen, Tong Xiao, Chunliang Zhang, and Jingbo Zhu. 2023a. Rethinking and improving multi-task learning for end-to-end speech translation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10753–10765, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yuhao+Zhang%2C+Chen+Xu%2C+Bei+Li%2C+Hao+Chen%2C+Tong+Xiao%2C+Chunliang+Zhang%2C+and+Jingbo+Zhu%2E+2023a%2E+Rethinking+and+improving+multi%2Dtask+learning+for+end%2Dto%2Dend+speech+translation%2E+In+Proceedings+of+the+2023+Co)
50. Zhiyang Zhang, Yaping Zhang, Yupu Liang, Lu Xiang, Yang Zhao, Yu Zhou, and Chengqing Zong. 2023b. LayoutDIT: Layout-aware end-to-end document image translation with multi-step conductive decoder. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10043–10053, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zhiyang+Zhang%2C+Yaping+Zhang%2C+Yupu+Liang%2C+Lu+Xiang%2C+Yang+Zhao%2C+Yu+Zhou%2C+and+Chengqing+Zong%2E+2023b%2E+LayoutDIT%3A+Layout%2Daware+end%2Dto%2Dend+document+image+translation+with+multi%2Dstep+conductive+decoder%2E+In+F)
51. Zhuosheng Zhang, Kehai Chen, Rui Wang, Masao Utiyama, Eiichiro Sumita, Zuchao Li, and Hai Zhao. 2020. Neural machine translation with universal visual representation. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhuosheng+Zhang%2C+Kehai+Chen%2C+Rui+Wang%2C+Masao+Utiyama%2C+Eiichiro+Sumita%2C+Zuchao+Li%2C+and+Hai+Zhao%2E+2020%2E+Neural+machine+translation+with+universal+visual+representation%2E+In+International+Conference+on+Le)
52. Shaolin Zhu, Shangjie Li, Yikun Lei, and Deyi Xiong. 2023a. PEIT: Bridging the modality gap with pre-trained models for end-to-end image translation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13433–13447, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shaolin+Zhu%2C+Shangjie+Li%2C+Yikun+Lei%2C+and+Deyi+Xiong%2E+2023a%2E+PEIT%3A+Bridging+the+modality+gap+with+pre%2Dtrained+models+for+end%2Dto%2Dend+image+translation%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+A) [our card](https://md.research.elamur.ai/free/papers/zhu-2023-peit-bridging-modality-gap-pre/)
53. Yaoming Zhu, Zewei Sun, Shanbo Cheng, Luyang Huang, Liwei Wu, and Mingxuan Wang. 2023b. Beyond triplet: Leveraging the most data for multimodal machine translation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 2679–2697, Toronto, Canada. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2212.10313)