---
title: "A Critical Study of Automatic Evaluation in — Yazdani 2026"
description: "Sign language translation (SLT) systems are usually scored only by comparing their output text with reference text."
authors: ["Shakib Yazdani", "Yasser HAMIDULLAH", "Cristina España-Bonet", "Eleftherios Avramidis", "Josef van Genabith"]
year: 2026
venue: "Proceedings of the Fifteenth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2026.lrec-1.749/"
doi: "10.63317/4n2sooe4fb2i"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/yazdani-2026-critical-study-automatic-evaluation-sign/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/yazdani-2026-critical-study-automatic-evaluation-sign/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2026](https://md.research.elamur.ai/free/papers/2026/) / A Critical Study of Automatic Evaluation in Sign Language Translation

# A Critical Study of Automatic Evaluation in Sign Language Translation

**Authors:** Shakib Yazdani, Yasser HAMIDULLAH, [Cristina España-Bonet](https://md.research.elamur.ai/free/authors/cristina-espana-bonet/), [Eleftherios Avramidis](https://md.research.elamur.ai/free/authors/eleftherios-avramidis/), [Josef van Genabith](https://md.research.elamur.ai/free/authors/josef-van-genabith/) · **Year:** 2026 · **Venue:** Proceedings of the Fifteenth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.lrec-1.749/) · [PDF](https://aclanthology.org/2026.lrec-1.749.pdf) · [DOI 10.63317/4n2sooe4fb2i](https://doi.org/10.63317/4n2sooe4fb2i) · [OpenAlex](https://openalex.org/W7160085169)

**Citations:** 1 citations (openalex-snapshot · 2026-10-14) · 1 in our library, of the 140673 papers whose bibliographies we read

## Summary
Sign language translation (SLT) systems are usually scored only by comparing their output text with reference text. It is unclear how well such scores reflect real translation quality. This study tests six reference-based metrics: the lexical overlap metrics BLEU, chrF and ROUGE, the embedding-based BLEURT, and the LLM judges G-Eval and GEMBA. The authors applied them to outputs from four off-the-shelf SLT models on the Phoenix-2014T benchmark of German Sign Language (DGS) weather forecasts. The metrics were stress-tested in three settings: outputs rewritten by an LLM, outputs containing hallucinations, and sentences grouped by length.

Overlap-based scores dropped for rewordings that kept the meaning intact. BLEURT and the LLM judges tracked semantic equivalence better, but they appeared to favour LLM-generated paraphrases. Every metric could tell hallucinated from clean outputs. BLEU, however, overreacted to severe hallucinations, while BLEURT and the LLM judges were too lenient on minor hallucinations in fluent text. Model rankings also changed depending on the metric and sentence length. The authors argue that SLT needs multimodal evaluation that also considers the sign video.

## Key points
- Models evaluated: TwoStream-SLT (gloss-based) and three gloss-free systems, SEM-SLT, SpaMo and Signformer, all on Phoenix-2014T.
- Paraphrases were produced with GPT-4o-mini at temperature 0.7. For the overlap metrics and BLEURT, a multi-reference variant was also used, with 11 references: the original plus 10 LLM-made paraphrases.
- Hallucinations were labelled by Llama-3-70B in a reference-based setup. It assigned four severity levels: no, small, partial or full hallucination.
- G-Eval rated adequacy and fluency on a 1–5 scale. GEMBA gave a zero-shot direct-assessment score from 0 to 100.
- Code and prompts are publicly available at github.com/shakibyzn/critical-eval-slt.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yazdani2026critical,
  title = {{A Critical Study of Automatic Evaluation in Sign Language Translation}},
  author = {Yazdani, Shakib and HAMIDULLAH, Yasser and España-Bonet, Cristina and Avramidis, Eleftherios and Genabith, Josef van},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4n2sooe4fb2i},
  url = {https://doi.org/10.63317/4n2sooe4fb2i}
}
```

- [← previous A Corpus-Based Profiling of Regional English Variants in Global Media: Insights from Olympic Journalism](https://md.research.elamur.ai/free/papers/mao-2026-corpus-based-profiling-regional-english/)
- [next → A Curious Class of Adpositional Multiword Expressions in Korean](https://md.research.elamur.ai/free/papers/min-2026-curious-class-adpositional-multiword-expressions/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/papers/2026/)

## References

46 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Kenza Benkirane, Laura Gongas, Shahar Pelles, Naomi Fuchs, Joshua Darmon, Pontus Stenetorp, David Ifeoluwa Adelani, and Eduardo Sánchez. 2024. Machine translation hallucination detection for low and high resource languages using large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9647–9665, Miami, Florida, USA. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2407.16470)
2. Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. 2018. Neural sign language translation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7784–7793. [find](https://scholar.google.com/scholar?q=Necati+Cihan+Camgoz%2C+Simon+Hadfield%2C+Oscar+Koller%2C+Hermann+Ney%2C+and+Richard+Bowden%2E+2018%2E+Neural+sign+language+translation%2E+In+2018+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+page)
3. Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. 2020. Sign language transformers: Joint end-to-end sign language recognition and translation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). [find](https://scholar.google.com/scholar?q=Necati+Cihan+Camgoz%2C+Oscar+Koller%2C+Simon+Hadfield%2C+and+Richard+Bowden%2E+2020%2E+Sign+language+transformers%3A+Joint+end%2Dto%2Dend+sign+language+recognition+and+translation%2E+In+IEEE+Conference+on+Computer+Visi)
4. Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. 2022a. A simple multi-modality transfer learning baseline for sign language translation. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5110–5120. [↗](https://doi.org/10.48550/arxiv.2203.04287)
5. Yutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu, Shujie Liu, and Brian Mak. 2022b. Two-stream network for sign language recognition and translation. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc. [find](https://scholar.google.com/scholar?q=Yutong+Chen%2C+Ronglai+Zuo%2C+Fangyun+Wei%2C+Yu+Wu%2C+Shujie+Liu%2C+and+Brian+Mak%2E+2022b%2E+Two%2Dstream+network+for+sign+language+recognition+and+translation%2E+In+Proceedings+of+the+36th+International+Conference+on)
6. Zhigang Chen, Benjia Zhou, Jun Li, Jun Wan, Zhen Lei, Ning Jiang, Quan Lu, and Guoqing Zhao. 2024. Factorized learning assisted with large language model for gloss-free sign language translation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 7071–7081, Torino, Italia. ELRA and ICCL. [find](https://scholar.google.com/scholar?q=Zhigang+Chen%2C+Benjia+Zhou%2C+Jun+Li%2C+Jun+Wan%2C+Zhen+Lei%2C+Ning+Jiang%2C+Quan+Lu%2C+and+Guoqing+Zhao%2E+2024%2E+Factorized+learning+assisted+with+large+language+model+for+gloss%2Dfree+sign+language+translation%2E+In+P)
7. Yiting Cheng, Fangyun Wei, Bao Jianmin, Dong Chen, and Wen Qiang Zhang. 2023. Cico: Domain-aware sign language retrieval via cross-lingual contrastive learning. In CVPR. [find](https://scholar.google.com/scholar?q=Yiting+Cheng%2C+Fangyun+Wei%2C+Bao+Jianmin%2C+Dong+Chen%2C+and+Wen+Qiang+Zhang%2E+2023%2E+Cico%3A+Domain%2Daware+sign+language+retrieval+via+cross%2Dlingual+contrastive+learning%2E+In+CVPR%2E)
8. OpenAI et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. [↗](https://arxiv.org/abs/2410.21276) [our card](https://md.research.elamur.ai/free/papers/openai-2024-gpt-4o-system-card/)
9. Marina Fomicheva, Shuo Sun, Erick Fonseca, Chrysoula Zerva, Frédéric Blain, Vishrav Chaudhary, Francisco Guzmán, Nina Lopatina, Lucia Specia, and André F. T. Martins. 2022. MLQE-PE: A multilingual quality estimation and post-editing dataset. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4963–4974, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Marina+Fomicheva%2C+Shuo+Sun%2C+Erick+Fonseca%2C+Chrysoula+Zerva%2C+Fr%C3%A9d%C3%A9ric+Blain%2C+Vishrav+Chaudhary%2C+Francisco+Guzm%C3%A1n%2C+Nina+Lopatina%2C+Lucia+Specia%2C+and+Andr%C3%A9+F%2E+T%2E+Martins%2E+2022%2E+MLQE%2DPE%3A+A+multilingual+qua) [our card](https://md.research.elamur.ai/free/papers/fomicheva-2022-mlqe-pe-multilingual-quality-estimation/)
10. Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frederic Blain, Tom Kocmi, Jiayi Wang, David Ifeoluwa Adelani, Marianna Buchicchio, Chrysoula Zerva, and Alon Lavie. 2024. Are LLMs breaking MT metrics? results of the WMT24 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, pages 47–81, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Markus+Freitag%2C+Nitika+Mathur%2C+Daniel+Deutsch%2C+Chi%2DKiu+Lo%2C+Eleftherios+Avramidis%2C+Ricardo+Rei%2C+Brian+Thompson%2C+Frederic+Blain%2C+Tom+Kocmi%2C+Jiayi+Wang%2C+David+Ifeoluwa+Adelani%2C+Marianna+Buchicchio%2C+Chrys) [our card](https://md.research.elamur.ai/free/papers/freitag-2024-are-llms-breaking-mt-metrics/)
11. Jia Gong, Lin Geng Foo, Yixuan He, Hossein Rahmani, and Jun Liu. 2024. Llms are good sign language translators. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18362– 18372. [find](https://scholar.google.com/scholar?q=Jia+Gong%2C+Lin+Geng+Foo%2C+Yixuan+He%2C+Hossein+Rahmani%2C+and+Jun+Liu%2E+2024%2E+Llms+are+good+sign+language+translators%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CV)
12. Mo Guan, Yan Wang, Guangkun Ma, Jiarui Liu, and Mingzu Sun. 2025. Mska: Multi-stream keypoint attention network for sign language recognition and translation. Pattern Recogn., 165(C). [↗](https://doi.org/10.1016/j.patcog.2025.111602)
13. Nuno M. Guerreiro, Duarte M. Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. 2023. Hallucinations in large multilingual translation models. Transactions of the Association for Computational Linguistics, 11:1500–1517. [↗](https://doi.org/10.1162/tacl_a_00615) [our card](https://md.research.elamur.ai/free/papers/guerreiro-2023-hallucinations-large-multilingual-translation-models/)
14. Yasser HAMIDULLAH, Koel Dutta Chowdhury, Yusser Al Ghussin, Shakib Yazdani, Cennet Oguz, Josef van Genabith, and Cristina España-Bonet. 2026. Grounding or guessing? visual signals for detecting hallucinations in sign language translation. In The Fourteenth International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.2510.18439)
15. Yasser Hamidullah, Josef van Genabith, and Cristina España-Bonet. 2024. Sign language translation with sentence embedding supervision. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 425–434, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yasser+Hamidullah%2C+Josef+van+Genabith%2C+and+Cristina+Espa%C3%B1a%2DBonet%2E+2024%2E+Sign+language+translation+with+sentence+embedding+supervision%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Association+for+)
16. Yasser Hamidullah, Shakib Yazdani, Cennet Oguz, Josef Van Genabith, and Cristina España-Bonet. 2025. SONAR-SLT: Multilingual sign language translation via language-agnostic sentence embedding supervision. In Proceedings of the Tenth Conference on Machine Translation, pages 301– 313, Suzhou, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yasser+Hamidullah%2C+Shakib+Yazdani%2C+Cennet+Oguz%2C+Josef+Van+Genabith%2C+and+Cristina+Espa%C3%B1a%2DBonet%2E+2025%2E+SONAR%2DSLT%3A+Multilingual+sign+language+translation+via+language%2Dagnostic+sentence+embedding+supervis)
17. Eui Jun Hwang, Sukmin Cho, Junmyeong Lee, and Jong C. Park. 2025. An efficient gloss-free sign language translation using spatial configurations and motion dynamics with LLMs. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3901– 3920, Albuquerque, New Mexico. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Eui+Jun+Hwang%2C+Sukmin+Cho%2C+Junmyeong+Lee%2C+and+Jong+C%2E+Park%2E+2025%2E+An+efficient+gloss%2Dfree+sign+language+translation+using+spatial+configurations+and+motion+dynamics+with+LLMs%2E+In+Proceedings+of+the+20)
18. Saki Imai, Mert Inan, Anthony B. Sicilia, and Malihe Alikhani. 2025. Silverscore: Semantically-aware embeddings for sign language generation evaluation. In Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing - Natural Language Processing in the Generative AI era, pages 452–461, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria. [↗](https://doi.org/10.48550/arxiv.2509.03791)
19. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12). [↗](https://doi.org/10.1145/3571730) [our card](https://md.research.elamur.ai/free/papers/ji-2022-survey-hallucination-natural-language-generation/)
20. Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Apostol Natsev, Mustafa Suleyman, and Andrew Zisserman. 2017. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950. [↗](https://arxiv.org/abs/1705.06950)
21. Jung-Ho Kim, Mathew Huerta-Enochian, Changyong Ko, and Du Hui Lee. 2024. SignBLEU: Automatic evaluation of multi-channel sign language translation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 14796–14811, Torino, Italia. ELRA and ICCL. [find](https://scholar.google.com/scholar?q=Jung%2DHo+Kim%2C+Mathew+Huerta%2DEnochian%2C+Changyong+Ko%2C+and+Du+Hui+Lee%2E+2024%2E+SignBLEU%3A+Automatic+evaluation+of+multi%2Dchannel+sign+language+translation%2E+In+Proceedings+of+the+2024+Joint+International+Confe)
22. Tom Kocmi and Christian Federmann. 2023. GEMBA-MQM: Detecting translation quality error spans with GPT-4. In Proceedings of the Eighth Conference on Machine Translation, pages 768– 775, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Christian+Federmann%2E+2023%2E+GEMBA%2DMQM%3A+Detecting+translation+quality+error+spans+with+GPT%2D4%2E+In+Proceedings+of+the+Eighth+Conference+on+Machine+Translation%2C+pages+768%E2%80%93+775%2C+Singapore%2E+Ass) [our card](https://md.research.elamur.ai/free/papers/kocmi-2023-gemba-mqm-detecting-translation-quality/)
23. Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li. 2020. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In The IEEE Winter Conference on Applications of Computer Vision, pages 1459–1469. [find](https://scholar.google.com/scholar?q=Dongxu+Li%2C+Cristian+Rodriguez%2C+Xin+Yu%2C+and+Hongdong+Li%2E+2020%2E+Word%2Dlevel+deep+sign+language+recognition+from+video%3A+A+new+large%2Dscale+dataset+and+methods+comparison%2E+In+The+IEEE+Winter+Conference+on+A)
24. Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of Psychology, 22(140):1–55. [find](https://scholar.google.com/scholar?q=Rensis+Likert%2E+1932%2E+A+technique+for+the+measurement+of+attitudes%2E+Archives+of+Psychology%2C+22%28140%29%3A1%E2%80%9355%2E)
25. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+ROUGE%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+Summarization+Branches+Out%2C+pages+74%E2%80%9381%2C+Barcelona%2C+Spain%2E+Association+for+Computational+Linguistics%2E)
26. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yang+Liu%2C+Dan+Iter%2C+Yichong+Xu%2C+Shuohang+Wang%2C+Ruochen+Xu%2C+and+Chenguang+Zhu%2E+2023%2E+G%2Deval%3A+NLG+evaluation+using+gpt%2D4+with+better+human+alignment%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+M)
27. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984–4997, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nitika+Mathur%2C+Timothy+Baldwin%2C+and+Trevor+Cohn%2E+2020%2E+Tangled+up+in+BLEU%3A+Reevaluating+the+evaluation+of+automatic+machine+translation+evaluation+metrics%2E+In+Proceedings+of+the+58th+Annual+Meeting+of) [our card](https://md.research.elamur.ai/free/papers/mathur-2020-tangled-up-bleu-reevaluating-evaluation/)
28. Mathias Müller, Malihe Alikhani, Eleftherios Avramidis, Richard Bowden, Annelies Braffort, Necati Cihan Camgöz, Sarah Ebling, Cristina España-Bonet, Anne Göhring, Roman Grundkiewicz, Mert Inan, Zifan Jiang, Oscar Koller, Amit Moryossef, Annette Rios, Dimitar Shterionov, Sandra Sidler-Miserez, Katja Tissi, and Davy Van Landuyt. 2023. Findings of the second WMT shared task on sign language translation (WMT-SLT23). In Proceedings of the Eighth Conference on Machine Translation, pages 68–94, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mathias+M%C3%BCller%2C+Malihe+Alikhani%2C+Eleftherios+Avramidis%2C+Richard+Bowden%2C+Annelies+Braffort%2C+Necati+Cihan+Camg%C3%B6z%2C+Sarah+Ebling%2C+Cristina+Espa%C3%B1a%2DBonet%2C+Anne+G%C3%B6hring%2C+Roman+Grundkiewicz%2C+Mert+Inan%2C+Zifan+) [our card](https://md.research.elamur.ai/free/papers/muller-2023-findings-second-wmt-shared-task/)
29. Mathias Müller, Sarah Ebling, Eleftherios Avramidis, Alessia Battisti, Michèle Berger, Richard Bowden, Annelies Braffort, Necati Cihan Camgöz, Cristina España-Bonet, Roman Grundkiewicz, Zifan Jiang, Oscar Koller, Amit Moryossef, Regula Perrollaz, Sabine Reinhard, Annette Rios, Dimitar Shterionov, Sandra Sidler-Miserez, Katja Tissi, and Davy Van Landuyt. 2022. Findings of the First WMT Shared Task on Sign Language Translation (WMT-SLT22). In Proceedings of the Seventh Conference on Machine Translation, pages 744–772, Abu Dhabi. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mathias+M%C3%BCller%2C+Sarah+Ebling%2C+Eleftherios+Avramidis%2C+Alessia+Battisti%2C+Mich%C3%A8le+Berger%2C+Richard+Bowden%2C+Annelies+Braffort%2C+Necati+Cihan+Camg%C3%B6z%2C+Cristina+Espa%C3%B1a%2DBonet%2C+Roman+Grundkiewicz%2C+Zifan+Jiang%2C+O)
30. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. [↗](https://doi.org/10.3115/1073083.1073135) [our card](https://md.research.elamur.ai/free/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
31. Maja Popović. 2015. chrF: character n-gram Fscore for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maja+Popovi%C4%87%2E+2015%2E+chrF%3A+character+n%2Dgram+Fscore+for+automatic+MT+evaluation%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages+392%E2%80%93395%2E+Association+for+Computational+Lin)
32. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/w18-6319) [our card](https://md.research.elamur.ai/free/papers/post-2018-call-clarity-reporting-bleu-scores/)
33. Ayako Sato, Kyotaro Nakajima, Hwichan Kim, Zhousi Chen, and Mamoru Komachi. 2024. TMU-HIT’s submission for the WMT24 quality estimation shared task: Is GPT-4 a good evaluator for machine translation? In Proceedings of the Ninth Conference on Machine Translation, pages 529– 534, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ayako+Sato%2C+Kyotaro+Nakajima%2C+Hwichan+Kim%2C+Zhousi+Chen%2C+and+Mamoru+Komachi%2E+2024%2E+TMU%2DHIT%E2%80%99s+submission+for+the+WMT24+quality+estimation+shared+task%3A+Is+GPT%2D4+a+good+evaluator+for+machine+translation%3F+)
34. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Dipanjan+Das%2C+and+Ankur+Parikh%2E+2020%2E+BLEURT%3A+Learning+robust+metrics+for+text+generation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+)
35. William C. Stokoe. 1980. Sign language structure. Annual Review of Anthropology, 9:365–390. [↗](https://doi.org/10.1146/annurev.an.09.100180.002053)
36. Sihan Tan, Taro Miyazaki, and Kazuhiro Nakadai. 2025. Multilingual gloss-free sign language translation: Towards building a sign language foundation model. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 553– 561, Vienna, Austria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sihan+Tan%2C+Taro+Miyazaki%2C+and+Kazuhiro+Nakadai%2E+2025%2E+Multilingual+gloss%2Dfree+sign+language+translation%3A+Towards+building+a+sign+language+foundation+model%2E+In+Proceedings+of+the+63rd+Annual+Meeting+of)
37. Tony Cheng Tong, Sirui He, Zhiwen Shao, and Dit-Yan Yeung. 2025. G-veval: A versatile metric for evaluating image and video captions using gpt-4o. Proceedings of the AAAI Conference on Artificial Intelligence, 39(7):7419–7427. [find](https://scholar.google.com/scholar?q=Tony+Cheng+Tong%2C+Sirui+He%2C+Zhiwen+Shao%2C+and+Dit%2DYan+Yeung%2E+2025%2E+G%2Dveval%3A+A+versatile+metric+for+evaluating+image+and+video+captions+using+gpt%2D4o%2E+Proceedings+of+the+AAAI+Conference+on+Artificial+Inte)
38. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824– 24837. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Maarten+Bosma%2C+brian+ichter%2C+Fei+Xia%2C+Ed+Chi%2C+Quoc+V+Le%2C+and+Denny+Zhou%2E+2022%2E+Chain%2Dof%2Dthought+prompting+elicits+reasoning+in+large+language+models%2E+In+Advanc) [our card](https://md.research.elamur.ai/free/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
39. Ryan Wong, Necati Cihan Camgoz, and Richard Bowden. 2024. Sign2GPT: Leveraging large language models for gloss-free sign language translation. In The Twelfth International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.2405.04164)
40. Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Wang. 2024. Pride and prejudice: LLM amplifies self-bias in self-refinement. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15474– 15492, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wenda+Xu%2C+Guanglei+Zhu%2C+Xuandong+Zhao%2C+Liangming+Pan%2C+Lei+Li%2C+and+William+Wang%2E+2024%2E+Pride+and+prejudice%3A+LLM+amplifies+self%2Dbias+in+self%2Drefinement%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+)
41. Eta Yang. 2024. Signformer is all you need: Towards edge ai for sign language. arXiv preprint arXiv:2411.12901. [↗](https://arxiv.org/abs/2411.12901)
42. Shakib Yazdani, Josef Van Genabith, and Cristina España-Bonet. 2025. Continual learning in multilingual sign language translation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 10923– 10938, Albuquerque, New Mexico. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shakib+Yazdani%2C+Josef+Van+Genabith%2C+and+Cristina+Espa%C3%B1a%2DBonet%2E+2025%2E+Continual+learning+in+multilingual+sign+language+translation%2E+In+Proceedings+of+the+2025+Conference+of+the+Nations+of+the+Americas+)
43. Aoxiong Yin, Zhou Zhao, Weike Jin, Meng Zhang, Xingshan Zeng, and Xiaofei He. 2022. Mlslt: Towards multilingual sign language translation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5099– 5109. [find](https://scholar.google.com/scholar?q=Aoxiong+Yin%2C+Zhou+Zhao%2C+Weike+Jin%2C+Meng+Zhang%2C+Xingshan+Zeng%2C+and+Xiaofei+He%2E+2022%2E+Mlslt%3A+Towards+multilingual+sign+language+translation%2E+In+2022+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Re)
44. Xianfeng Zeng, Yijin Liu, Fandong Meng, and Jie Zhou. 2024. Towards multiple references era – addressing data leakage and limited reference diversity in machine translation evaluation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 11939–11951, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xianfeng+Zeng%2C+Yijin+Liu%2C+Fandong+Meng%2C+and+Jie+Zhou%2E+2024%2E+Towards+multiple+references+era+%E2%80%93+addressing+data+leakage+and+limited+reference+diversity+in+machine+translation+evaluation%2E+In+Findings+of+)
45. Biao Zhang, Mathias Müller, and Rico Sennrich. 2023. SLTUNET: A simple unified model for sign language translation. In The Eleventh International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.2305.01778)
46. Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. 2021. Improving sign language translation with monolingual data by sign back-translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1316–1325. [find](https://scholar.google.com/scholar?q=Hao+Zhou%2C+Wengang+Zhou%2C+Weizhen+Qi%2C+Junfu+Pu%2C+and+Houqiang+Li%2E+2021%2E+Improving+sign+language+translation+with+monolingual+data+by+sign+back%2Dtranslation%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Co)