--- title: "Evaluating the Performance of Transformers in Translating" description: "Akkadian is an extinct East Semitic language. 1 reference. Among them: 0." authors: ["Daniel A. Jones", "Ruslan Mitkov"] year: 2025 venue: "Proceedings of the First Workshop on Comparative Performance Evaluation: From Rules to Language Models" source: "https://aclanthology.org/2025.r2lm-1.5/" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/jones-2025-evaluating-performance-transformers-translating-low/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jones-2025-evaluating-performance-transformers-translating-low/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/4/) / Evaluating the Performance of Transformers in Translating Low-Resource Languages through Akkadian # Evaluating the Performance of Transformers in Translating Low-Resource Languages through Akkadian **Authors:** Daniel A. Jones and [Ruslan Mitkov](https://md.research.elamur.ai/free/en/authors/ruslan-mitkov/) · **Year:** 2025 · **Venue:** Proceedings of the First Workshop on Comparative Performance Evaluation: From Rules to Language Models · **Publisher:** [INCOMA Ltd.](https://md.research.elamur.ai/free/en/publishers/incoma-ltd/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.r2lm-1.5/) · [PDF](https://aclanthology.org/2025.r2lm-1.5.pdf) · [OpenAlex](https://openalex.org/W7131662888) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Akkadian is an extinct East Semitic language. Huge numbers of its cuneiform tablets have been transliterated, but few have been translated. Unlike most low-resource languages, it has a large, carefully annotated parallel corpus. That makes it a convenient laboratory for comparing translation architectures without heavy data augmentation. The authors collected transliterated Akkadian–English pairs from Oracc and CDLI and normalised mixed ASCII/Unicode transliteration conventions. They then fine-tuned several pretrained models on consumer hardware using quantisation and LoRA: - seq2seq models: MarianMT variants initialised from Arabic→English and Spanish→English, and T5 - decoder-only LLMs: Mistral-7B and Qwen-0.5B Krueger's published T5 served as an external reference. Evaluation used BLEU, ROUGE-2, BERTScore, pairwise significance tests, length-bucketed analysis and inference timing. The Arabic-initialised MarianMT gave the best overall balance and was by far the fastest. Mistral-7B edged it on BLEU but was much slower. The Spanish-initialised Marian performed far worse, which points to real transfer value from a related language. The authors caution that repetition in the corpus inflates absolute scores. ## Key points - **Translation gap:** in the Ur III corpus, roughly 72,000 texts are transcribed, yet only about 2.2% have modern translations. - **BERTScore results:** - MarianAr scored 0.931, and Mistral, Qwen and Krueger's T5 scored about 0.930. - The authors' own T5 (15 epochs) reached 0.914, and MarianEs only 0.842. - **Short references (\<4 words):** the LLMs did best on BLEU. Mistral reached 0.602, Qwen 0.593 and Krueger 0.586, against 0.541 for MarianAr. - **Speed (500 sentences):** | Model | Total time | Per sentence | |---|---|---| | MarianAr | 108 s | 0.22 s | | T5 | 242 s | 0.48 s | | Qwen | 348 s | 0.70 s | | Mistral | 1786 s | 3.57 s | - **Significance and training notes:** all pairwise BLEU differences had p \< 0.05. Mistral-7B was fine-tuned for just one epoch and Qwen-0.5B for three. Krueger's T5 outperformed the authors' 15-epoch T5, which the paper credits to his 30 epochs and bidirectional training. ## Interesting ideas - Picking the pretrained starting point by linguistic kinship (Arabic rather than Spanish) changed results dramatically, so the choice of initialisation deserves deliberate testing as a variable. - Splitting evaluation by reference length showed that big LLMs shine mainly on very short segments, a pattern that a single averaged score hides. - The authors openly flag that near-duplicate phrases occur in both train and test splits and inflate scores, while noting that model comparisons remain valid. - Measuring wall-clock inference cost next to quality made the compact model the practical winner despite a slightly lower BLEU. - Training a model to translate in both directions (source→English and back) was reported to stabilise convergence for a small seq2seq model. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (tree): When adapting gparse to a new UD dialect or genre, compare initialising from different source treebanks and keep the one closest in structure, not just the largest. - **I2** (learn): Report parser and reader accuracy per sentence-length bucket, so gains limited to short sentences are not mistaken for general improvement. - **I3** (learn): Add a near-duplicate check between silver training data and test sets as a standard negative control before trusting absolute numbers. - **I4** (tree): Publish sentences-per-second alongside LAS when comparing the perceptron parser and gparse, so pipeline choices reflect the real speed–accuracy trade-off. - **I5** (back): Train the text→graph and graph→text halves of the compressor jointly, so each direction regularises the other. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Alicante](https://md.research.elamur.ai/free/en/universities/university-of-alicante/) — University, Alicante, Spain · 14 works · 1,081 citations - [Lancaster University](https://md.research.elamur.ai/free/en/universities/lancaster-university/) — University, Lancaster, United Kingdom · 16 works · 1,153 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works 🧭 Topics (4): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 1,485 works · [🌱 Low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/) · 165 works · [𒀭 Akkadian and cuneiform NLP](https://md.research.elamur.ai/free/en/topics/akkadian-cuneiform-nlp/) · 25 works ## Cite Cite as: Daniel A. Jones and Ruslan Mitkov. 2025. Evaluating the Performance of Transformers in Translating Low-Resource Languages through Akkadian. Proceedings of the First Workshop on Comparative Performance Evaluation: From Rules to Language Models. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{jones2025evaluating, title = {{Evaluating the Performance of Transformers in Translating Low-Resource Languages through Akkadian}}, author = {Daniel A. Jones and Ruslan Mitkov}, booktitle = {Proceedings of the First Workshop on Comparative Performance Evaluation: From Rules to Language Models}, year = {2025}, doi = {10.26615/978-954-452-102-8-005}, url = {https://doi.org/10.26615/978-954-452-102-8-005} } ``` - [← previous Enhancing Quranic Learning: A Multimodal Deep Learning Approach for Arabic Phoneme Recognition](https://md.research.elamur.ai/free/en/papers/kucukmanisa-2025-enhancing-quranic-learning-multimodal-deep/) - [next → Evaluation on Chinese Multiturn Instruction Following Capability of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhu-2025-evaluation-chinese-multiturn/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/4/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Ahmed M. Agiza, Kai Zhu, Tianlong Zhang, and Shijie Han. 2024. Mtlora: Low-rank adaptation approach for efficient multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16196–16205. [find](https://scholar.google.com/scholar?q=Ahmed+M%2E+Agiza%2C+Kai+Zhu%2C+Tianlong+Zhang%2C+and+Shijie+Han%2E+2024%2E+Mtlora%3A+Low%2Drank+adaptation+approach+for+efficient+multi%2Dtask+learning%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+) *Updated 24 October 2026*