---
title: "UCCIX: Irish-eXcellence Large Language Model (2024)"
description: "The authors build UCCIX, an open-source Irish LLM, by continued pre-training of Llama 2-13B. 14 references, cited by 12 papers here."
authors: ["Khanh-Tung Tran", "Barry O'Sullivan", "Hoang D. Nguyen"]
year: 2024
venue: "arXiv:2405.13010"
source: "https://arxiv.org/abs/2405.13010"
doi: "10.48550/arxiv.2405.13010"
arxiv: "2405.13010"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tran-2024-uccix-irish-excellence-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tran-2024-uccix-irish-excellence-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/11/) / UCCIX: Irish-eXcellence Large Language Model

# UCCIX: Irish-eXcellence Large Language Model

**Authors:** [Khanh-Tung Tran](https://md.research.elamur.ai/free/en/authors/khanh-tung-tran/), [Barry O'Sullivan](https://md.research.elamur.ai/free/en/authors/barry-o-sullivan/) and [Hoang D. Nguyen](https://md.research.elamur.ai/free/en/authors/hoang-d-nguyen/) · **Year:** 2024 · **Venue:** [arXiv:2405.13010](https://md.research.elamur.ai/free/en/venues/frontiers-in-artificial-intelligence-and-applications/) ([Frontiers in artificial intelligence and applications 2024](https://md.research.elamur.ai/free/en/venues/frontiers-in-artificial-intelligence-and-applications/2024/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2405.13010) · [PDF](https://arxiv.org/pdf/2405.13010) · [DOI 10.48550/arxiv.2405.13010](https://doi.org/10.48550/arxiv.2405.13010) · [arXiv 2405.13010](https://arxiv.org/abs/2405.13010) · [OpenAlex](https://openalex.org/W4403479290)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [12](https://md.research.elamur.ai/free/en/papers/tran-2024-uccix-irish-excellence-large-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Irish is a UNESCO "definitely endangered" language with only a few hundred million tokens of available text, far below what scaling laws suggest for training an LLM. The authors build UCCIX, an open-source Irish LLM, by continued pre-training of Llama 2-13B. They selected that base by its perplexity on Irish. They gathered and cleaned the Irish open-source data they could find, removed duplicates across sources with 5-gram matching, and added 10k Irish tokens to the tokenizer. Training first used English–Irish parallel sentences, then a much larger monolingual Irish corpus. An instruction-tuned variant, UCCIX-Instruct, followed. Because Irish lacks generative benchmarks, they also built IrishQA and an Irish version of MT-bench. UCCIX beats much larger models on Irish tasks, by up to 12 points on the Cloze Test. Its performance on English benchmarks drops, which the authors attribute to catastrophic forgetting. The work matters as a low-data recipe that could transfer to other endangered or indigenous languages, including other low-resource settings.

## Key points
- **Pre-training corpus:** about 3.1B characters before cleaning and 1.8B after. Sources are CulturaX-ga (65%), Glot500-ga (27.1%), the Irish side of ParaCrawl v7, Irish Wikipedia and Corpora Irish. ELRC parallel data, about 1% of the corpus size, is scheduled first.
- **Base model choice:** Llama 2-13B had the lowest Irish perplexity (8.94), ahead of Mistral-7B (11.68) and BLOOM-7B (63.75). Vocabulary grew from 32k to 42k tokens.
- **Irish results:** Cloze Test accuracy 0.75 vs 0.63 for Llama 2-70B; SIB-200 (Irish) 0.78 vs 0.71; IrishQA (ga) exact match 0.39 vs 0.31. Ablations show that both the parallel-data stage and the vocabulary expansion help.
- **Tokenizer efficiency:** about 3.47 chars/token vs 2.26 for Llama 2, roughly 1.54× faster generation. Fertility fell from 1.95 to 1.57 tokens/word.
- **New resources:** IrishQA has 60 context-grounded QA pairs in English and Irish, written by native speakers. MT-bench was machine-translated and then corrected by native speakers. On Irish MT-bench, UCCIX-Instruct scores 5.01, close to gpt-3.5-turbo's 5.47, while Llama 2-13B-chat scores 1.94.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University College Cork](https://md.research.elamur.ai/free/en/universities/university-college-cork/) — University, Cork, Ireland · 6 works · 119 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Khanh-Tung Tran, Barry O'Sullivan and Hoang D. Nguyen. 2024. UCCIX: Irish-eXcellence Large Language Model. arXiv:2405.13010.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{tran2024uccix,
  title = {{UCCIX: Irish-eXcellence Large Language Model}},
  author = {Khanh-Tung Tran, Barry O'Sullivan and Hoang D. Nguyen},
  year = {2024},
  doi = {10.48550/arxiv.2405.13010},
  eprint = {2405.13010},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.13010}
}
```

- [← previous Towards a dynamical model of English vowels. Evidence from diphthongisation](https://md.research.elamur.ai/free/en/papers/strycharczuk-2024-towards-dynamical-model-english-vowels/)
- [next → Understanding Chain-of-Thought in LLMs through Information Theory](https://md.research.elamur.ai/free/en/papers/ton-2024-understanding-chain-thought-llms-through/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/11/)

## References

14 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Cloud Translation — cloud.google.com. [https://cloud.google.com/](https://cloud.google.com/) translate?hl=en. [Accessed 09-05-2024]. [↗](https://cloud.google.com/)
2. David Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba Alabi, Yanke Mao, Haonan Gao, and En-Shiun Lee, ‘[SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects](https://md.research.elamur.ai/free/en/papers/adelani-2023-sib-200-simple-inclusive-big/)’, in Proceedings of the 18th Conference of the European Chapter of the…
3. Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck, ‘[Llemma: An open language model for mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/)’, in The Twelfth International Conference on Learning Representations, (2024).
4. Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel L. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Ramírez-Sánchez, Elsa Sarrías, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume… — [ParaCrawl: Web-Scale Acquisition of Parallel Corpora](https://md.research.elamur.ai/free/en/papers/banon-2020-paracrawl-web-scale-acquisition-parallel/)
5. James Barry, Joachim Wagner, Lauren Cassidy, Alan Cowap, Teresa Lynn, Abigail Walsh, Mícheál J. Ó Meachair, and Jennifer Foster, ‘gaBERT — an Irish language model’, in Proceedings of the Thirteenth Language Resources and Evaluation Conference, eds., Nicoletta Calzolari, Frédéric Béchet, Philippe Bla… [find](https://scholar.google.com/scholar?q=James+Barry%2C+Joachim+Wagner%2C+Lauren+Cassidy%2C+Alan+Cowap%2C+Teresa+Lynn%2C+Abigail+Walsh%2C+M%C3%ADche%C3%A1l+J%2E+%C3%93+Meachair%2C+and+Jennifer+Foster%2C+%E2%80%98gaBERT+%E2%80%94+an+Irish+language+model%E2%80%99%2C+in+Proceedings+of+the+Thirteenth+La)
6. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie, ‘[A survey on evaluation of large language models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/)’, ACM Trans. Intell. Syst. Technol., 15(3), (mar 2024).
7. European Language Resource Coordination, AI for Multilingual Europe – Why Language Data Matters, ELRC Consortium, 2022. [find](https://scholar.google.com/scholar?q=European+Language+Resource+Coordination%2C+AI+for+Multilingual+Europe+%E2%80%93+Why+Language+Data+Matters%2C+ELRC+Consortium%2C+2022%2E)
8. Leo Gao et al. A framework for few-shot language model evaluation, 12 2023. [find](https://scholar.google.com/scholar?q=Leo+Gao+et+al%2E+A+framework+for+few%2Dshot+language+model+evaluation%2C+12+2023%2E)
9. Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff, ‘[Building large monolingual dictionaries at the Leipzig corpora collection: From 100 to 200 languages](https://md.research.elamur.ai/free/en/papers/goldhahn-2012-building-large-monolingual-dictionaries-leipzig/)’, in Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), eds., Nicoletta Calzolari, Khalid Choukri, Th…
10. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi… [find](https://scholar.google.com/scholar?q=Suriya+Gunasekar%2C+Yi+Zhang%2C+Jyoti+Aneja%2C+Caio+C%C3%A9sar+Teodoro+Mendes%2C+Allie+Del+Giorno%2C+Sivakanth+Gopi%2C+Mojan+Javaheripi%2C+Piero+Kauffmann%2C+Gustavo+de+Rosa%2C+Olli+Saarikivi%2C+Adil+Salim%2C+Shital+Shah%2C+Harki)
11. Jordan Hoffmann et al., ‘An empirical analysis of compute-optimal large language model training’, in Advances in Neural Information Processing Systems, eds., Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, (2022). [find](https://scholar.google.com/scholar?q=Jordan+Hoffmann+et+al%2E%2C+%E2%80%98An+empirical+analysis+of+compute%2Doptimal+large+language+model+training%E2%80%99%2C+in+Advances+in+Neural+Information+Processing+Systems%2C+eds%2E%2C+Alice+H%2E+Oh%2C+Alekh+Agarwal%2C+Danielle+Belgr)
12. Ayyoob ImaniGooghari, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André Martins, François Yvon, and Hinrich Schütze, ‘Glot500: Scaling multilingual corpora and language models to 500 languages’, in Proceedings of the 61st Annual M… [find](https://scholar.google.com/scholar?q=Ayyoob+ImaniGooghari%2C+Peiqin+Lin%2C+Amir+Hossein+Kargaran%2C+Silvia+Severini%2C+Masoud+Jalili+Sabet%2C+Nora+Kassner%2C+Chunlan+Ma%2C+Helmut+Schmid%2C+Andr%C3%A9+Martins%2C+Fran%C3%A7ois+Yvon%2C+and+Hinrich+Sch%C3%BCtze%2C+%E2%80%98Glot500%3A+Sca)
13. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+Las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
14. Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed, ‘Mistral 7b’, CoRR, abs/2310.06825, (2023). Seamus Lankford, Haithem Afli, and Andy Way, ‘Machine translation in the covid domain: an English-Irish case study for LoResMT 2021’, in Proceedings of the 4th Workshop on Technologies for MT of… [find](https://scholar.google.com/scholar?q=Lavril%2C+Thomas+Wang%2C+Timoth%C3%A9e+Lacroix%2C+and+William+El+Sayed%2C+%E2%80%98Mistral+7b%E2%80%99%2C+CoRR%2C+abs%2F2310%2E06825%2C+%282023%29%2E+Seamus+Lankford%2C+Haithem+Afli%2C+and+Andy+Way%2C+%E2%80%98Machine+translation+in+the+covid+domain%3A+an+Engli)

*Updated 31 October 2026*