---
title: "The Helsinki-NLP Submissions at NADI 2023 Shared Task:"
description: "The Helsinki-NLP team entered the NADI 2023 shared task on translating four Arabic… 3.48 BLEU. 27 references, cited by 3 papers here."
authors: ["Yves Scherrer", "Aleksandra Miletić", "Olli Kuparinen"]
year: 2023
venue: "Proceedings of ArabicNLP 2023"
source: "https://aclanthology.org/2023.arabicnlp-1.73/"
doi: "10.18653/v1/2023.arabicnlp-1.73"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/scherrer-2023-helsinki-nlp-submissions-nadi-2023/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/scherrer-2023-helsinki-nlp-submissions-nadi-2023/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/) / The Helsinki-NLP Submissions at NADI 2023 Shared Task: Walking the Baseline

# The Helsinki-NLP Submissions at NADI 2023 Shared Task: Walking the Baseline

**Authors:** [Yves Scherrer](https://md.research.elamur.ai/free/en/authors/yves-scherrer/), Aleksandra Miletić and Olli Kuparinen · **Year:** 2023 · **Venue:** [Proceedings of ArabicNLP 2023](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2023](https://md.research.elamur.ai/free/en/venues/arabicnlp/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.arabicnlp-1.73/) · [PDF](https://aclanthology.org/2023.arabicnlp-1.73.pdf) · [DOI 10.18653/v1/2023.arabicnlp-1.73](https://doi.org/10.18653/v1/2023.arabicnlp-1.73) · [OpenAlex](https://openalex.org/W4389518338)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/scherrer-2023-helsinki-nlp-submissions-nadi-2023/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The Helsinki-NLP team entered the NADI 2023 shared task on translating four Arabic dialects (Egyptian, Emirati, Jordanian, Palestinian) into Modern Standard Arabic (MSA). They submitted seven systems across the closed track (MADAR only) and the open track (any data). They used a subset of the MADAR corpus covering the Nile Basin, Levant and Gulf regions (MADAR3) and compared four approaches:

- a copy-the-input baseline (LAI);
- character-level Moses SMT;
- a Transformer NMT;
- fine-tuned ByT5 and AraT5.

For the open track, they gathered large monolingual MSA corpora. These were used as extra SMT language models and, after backtranslation into dialects, as synthetic parallel data for the neural systems. Both tracks ranked them second.

In the closed track, no real system beat simply copying the input on BLEU. In the open track, character-level SMT was their best system, with AraT5 close behind. Neural models only caught up once synthetic data was added.

The paper's main lessons concern evaluation. A trivial baseline exposes how hard a task is, and a character-based metric (chrF) can reorder systems that BLEU ranks differently.

## Key points
- **Closed-track test BLEU:**
  - LAI (copy baseline): 14.28.
  - SMT-multi: 13.60.
  - SMT-mono: 12.53.
  - Best competitor: 14.76.
  - On Emirati, LAI scored 23.13 against the best competitor's 14.30, which suggests other systems over-normalize this dialect.
- **Open-track test BLEU:**
  - SMT-multi (seven MSA language models): 17.69.
  - AraT5 V2 (fine-tuned on 1.4M pairs including backtranslations): 17.46.
  - Best competitor: 21.10.
- **MSA corpora:** AraNews, Leipzig News, OSIAN, Tatoeba, TED and Arabic Wikipedia (about 11.4M sentences). Backtranslation was done by running the dialect-specific SMT-mono models in reverse.
- **NMT setup:** SentencePiece unigram segmentation with a very small vocabulary (300 worked best of 300/500/1000), plus a dialect tag prepended to the source sentence.
- **Metric disagreement on dev:** BLEU ranks LAI first. chrF ranks SMT-mono first on every dialect (46.96 overall versus 45.02 for LAI).
- **Domain gap:** On held-out MADAR3 test data, LAI drops to 3.48 BLEU. Open SMT-multi reaches 10.22 there, while AraT5 reaches 17.85, so AraT5 generalizes better to the harder domain.

## Interesting ideas
- Submitting an unchanged copy of the input as a system measures how much real work a normalization task demands, and shows when "improvements" are actually harmful edits.
- When the target is close to the source, treating characters as the translation units (in SMT, and through a tiny subword vocabulary in NMT) captures small spelling shifts better than word units do.
- A word-level metric like BLEU punishes one-letter errors as fully wrong, while chrF credits partial correctness, and the two metrics ranked the systems differently.
- Prepending a variety tag to the input lets one shared model serve several dialects. Separate per-dialect models were also compared with a single pooled one.
- Evaluating the same systems on an in-domain and an out-of-domain test set showed that systems tied on one set can differ greatly on the other.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Add an identity/no-op output as a mandatory negative control in every repair-rule or normalization evaluation, so a rule is only kept if it beats doing nothing.
- **I2** (tokens): For spelling variants, archaic or dialectal English forms, a character-level edit model mapping to standard forms could feed the lemmatizer before tagging.
- **I3** (learn): Report both a strict and a character-/partial-credit score for morphology and lemma outputs, and flag cases where the two rankings disagree.
- **I4** (tree): gparse could take a source-genre or treebank-dialect tag as an input token, so one parser handles several UD conventions without separate models.
- **I5** (answer): Test readers on both fables and school texts, so a tie on one set does not hide poor generalization.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Helsinki](https://md.research.elamur.ai/free/en/universities/university-of-helsinki/) — University, Helsinki, Finland · 50 works · 5,318 citations
- [University of Oslo](https://md.research.elamur.ai/free/en/universities/university-of-oslo/) — University, Oslo, Norway · 51 works · 4,155 citations

Also on the byline: [Tampere University](https://ror.org/033003e23) · [University of Tampere](https://ror.org/05p6qqk79).

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Yves Scherrer, Aleksandra Miletić and Olli Kuparinen. 2023. The Helsinki-NLP Submissions at NADI 2023 Shared Task: Walking the Baseline. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{scherrer2023helsinki,
  title = {{The Helsinki-NLP Submissions at NADI 2023 Shared Task: Walking the Baseline}},
  author = {Yves Scherrer, Aleksandra Miletić and Olli Kuparinen},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.73},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.73}
}
```

- [← previous The DARPA Wikidata Overlay: Wikidata as an ontology for natural language processing](https://md.research.elamur.ai/free/en/papers/spaulding-2023-darpa-wikidata-overlay-wikidata-as/)
- [next → Transformers and Ensemble methods: A solution for Hate Speech Detection in Arabic languages](https://md.research.elamur.ai/free/en/papers/paula-2023-transformers-ensemble-methods-solution-hate/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/)

## References

27 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, Chiyu Zhang, ElMoatez Billah Nagoudi, Houda Bouamor, and Nizar Habash. 2023. [NADI 2023: The Fourth Nuanced Arabic Dialect Identification Shared Task](https://md.research.elamur.ai/free/en/papers/mageed-2023-nadi-2023-fourth-nuanced-arabic/). In Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023).
2. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. [NADI 2020: The first nuanced Arabic dialect identification shared task](https://md.research.elamur.ai/free/en/papers/mageed-2020-nadi-2020-first-nuanced-arabic/). In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 97–110, Barcelona, Spain (Online). Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2010.11334)
3. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2021. [NADI 2021: The second nuanced Arabic dialect identification shared task](https://md.research.elamur.ai/free/en/papers/mageed-2021-nadi-2021-second-nuanced-arabic/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 244–259, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2103.08466)
4. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. [NADI 2022: The third nuanced Arabic dialect identification shared task](https://md.research.elamur.ai/free/en/papers/mageed-2022-nadi-2022-third-nuanced-arabic/). In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 85–97, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
5. Giuseppe Attardi. 2015. Wikiextractor. [https://github.com/attardi/wikiextractor](https://github.com/attardi/wikiextractor). [↗](https://github.com/attardi/wikiextractor)
6. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Houda+Bouamor%2C+Nizar+Habash%2C+Mohammad+Salameh%2C+Wajdi+Zaghouani%2C+Owen+Rambow%2C+Dana+Abdulrahim%2C+Ossama+Obeid%2C+Salam+Khalifa%2C+Fadhl+Eryani%2C+Alexander+Erdmann%2C+and+Kemal+Oflazer%2E+2018%2E+The+MADAR+Arabic+di)
7. Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff. 2012. [Building large monolingual dictionaries at the leipzig corpora collection: From 100 to 200 languages](https://md.research.elamur.ai/free/en/papers/goldhahn-2012-building-large-monolingual-dictionaries-leipzig/). In International Conference on Language Resources and Evaluation.
8. Taku Kudo and John Richardson. 2018. [SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
9. Olli Kuparinen, Aleksandra Miletić, and Yves Scherrer. 2023. Dialect-to-Standard Normalization: A Large-Scale Multilingual Evaluation. In Findings of EMNLP2023. (accepted). [find](https://scholar.google.com/scholar?q=Olli+Kuparinen%2C+Aleksandra+Miletic%CC%81%2C+and+Yves+Scherrer%2E+2023%2E+Dialect%2Dto%2DStandard+Normalization%3A+A+Large%2DScale+Multilingual+Evaluation%2E+In+Findings+of+EMNLP2023%2E+%28accepted%29%2E)
10. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. [AraT5: Text-to-text transformers for Arabic language generation](https://md.research.elamur.ai/free/en/papers/nagoudi-2022-arat5-text-text-transformers-arabic/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics.
11. El Moatez Billah Nagoudi, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Tariq Alhindi. 2020. [Machine generation and detection of Arabic manipulated and fake news](https://md.research.elamur.ai/free/en/papers/nagoudi-2020-machine-generation-detection-arabic-manipulated/). In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 69–84, Barcelona, Spain (Online). Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2011.03092)
12. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. [↗](https://doi.org/10.3115/1073083.1073135)
13. Maja Popović. 2015. [chrF: character n-gram F-score for automatic MT evaluation](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/). In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/w15-3049)
14. Nizar Y Habash. 2022. Introduction to [Arabic natural language processing](https://md.research.elamur.ai/free/en/papers/habash-2022-arabic-natural-language-processing/). Springer Nature.
15. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67. [↗](https://doi.org/10.48550/arxiv.1910.10683)
16. Vani Kanjirangat, Tanja Samardžić, Ljiljana Dolamic, and Fabio Rinaldi. 2023. [Optimizing the size of subword vocabularies in dialect classification](https://md.research.elamur.ai/free/en/papers/kanjirangat-2023-optimizing-size-subword-vocabularies-dialect/). In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), pages 14–30, Dubrovnik, Croatia. Association for Computational Linguistics.
17. Nils Reimers and Iryna Gurevych. 2020. [Making monolingual sentence embeddings multilingual using knowledge distillation](https://md.research.elamur.ai/free/en/papers/reimers-2020-making-monolingual-sentence-embeddings-multilingual/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
18. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. [OpenNMT: Open-source toolkit for neural machine translation](https://md.research.elamur.ai/free/en/papers/klein-2017-opennmt-open-source-toolkit-neural/). In Proceedings of ACL 2017, System Demonstrations, pages 67–72, Vancouver, Canada. Association for Computational Linguistics.
19. David Samuel and Milan Straka. 2021. ÚFAL at Multi-LexNorm 2021: Improving multilingual lexical normalization by fine-tuning ByT5. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 483–492, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+Samuel+and+Milan+Straka%2E+2021%2E+%C3%9AFAL+at+Multi%2DLexNorm+2021%3A+Improving+multilingual+lexical+normalization+by+fine%2Dtuning+ByT5%2E+In+Proceedings+of+the+Seventh+Workshop+on+Noisy+User%2Dgenerated+Text+%28)
20. Yves Scherrer. 2023. Character alignment methods for dialect-to-standard normalization. In Proceedings of the 20th SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 110–116, Toronto, Canada. Association for Computational Linguistics. [↗](https://doi.org/10.48448/774x-7874)
21. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016%2E+Improving+neural+machine+translation+models+with+monolingual+data%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computationa)
22. Toshiyuki Takezawa, Genichiro Kikui, Masahide Mizushima, and Eiichiro Sumita. 2007. Multilingual spoken language corpus development for communication research. In International Journal of Computational Linguistics & Chinese Language Processing, Volume 12, Number 3, September 2007: Special Issue on Invited Papers from ISCSLP 2006, pages 303–324. [find](https://scholar.google.com/scholar?q=Toshiyuki+Takezawa%2C+Genichiro+Kikui%2C+Masahide+Mizushima%2C+and+Eiichiro+Sumita%2E+2007%2E+Multilingual+spoken+language+corpus+development+for+communication+research%2E+In+International+Journal+of+Computationa)
23. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=J%C3%B6rg+Tiedemann%2E+2012%2E+Parallel+data%2C+tools+and+interfaces+in+opus%2E+In+Proceedings+of+the+Eight+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9912%29%2C+Istanbul%2C+Turkey%2E+European+Langu)
24. Rob van der Goot, Alan Ramponi, Arkaitz Zubiaga, Barbara Plank, Benjamin Muller, Iñaki San Vicente Roncal, Nikola Ljubešić, Özlem Çetinoğlu, Rahmad Mahendra, Talha Çolakoğlu, Timothy Baldwin, Tommaso Caselli, and Wladimir Sidorenko. 2021. MultiLexNorm: A shared task on multilingual lexical normalization. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 493–509, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rob+van+der+Goot%2C+Alan+Ramponi%2C+Arkaitz+Zubiaga%2C+Barbara+Plank%2C+Benjamin+Muller%2C+I%C3%B1aki+San+Vicente+Roncal%2C+Nikola+Ljube%C5%A1ic%CC%81%2C+%C3%96zlem+%C3%87etinog%CC%86lu%2C+Rahmad+Mahendra%2C+Talha+%C3%87olakog%CC%86lu%2C+Timothy+Baldwin%2C+Tomma)
25. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306. [↗](https://doi.org/10.1162/tacl_a_00461)
26. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. [mT5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2021.naacl-main.41)
27. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. [OSIAN: Open source international Arabic news corpus - preparation and integration into the CLARIN-infrastructure](https://md.research.elamur.ai/free/en/papers/zeroual-2019-osian-open-source-international-arabic/). In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 175–182, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.60692/vm65h-ach57)

*Updated 1 November 2026*