--- title: "Language Adaptation on a Tight Academic Compute Budget:" description: "The paper asks whether an academic lab with only a handful of server-grade GPUs and a few days of access can… 0.6 points. 2 references." authors: ["Konstantin Dobler", "Gerard de Melo"] year: 2024 venue: "arXiv:2408.15793" source: "https://arxiv.org/abs/2408.15793" doi: "10.48550/arxiv.2408.15793" arxiv: "2408.15793" licence: "CC BY-SA 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by-sa/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/dobler-2024-language-adaptation-tight-academic-compute/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dobler-2024-language-adaptation-tight-academic-compute/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/16/) / Language Adaptation on a Tight Academic Compute Budget: Tokenizer Swapping Works and Pure bfloat16 Is Enough # Language Adaptation on a Tight Academic Compute Budget: Tokenizer Swapping Works and Pure bfloat16 Is Enough **Authors:** [Konstantin Dobler](https://md.research.elamur.ai/free/en/authors/konstantin-dobler/) and [Gerard de Melo](https://md.research.elamur.ai/free/en/authors/gerard-de-melo/) · **Year:** 2024 · **Venue:** arXiv:2408.15793 · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2408.15793) · [PDF](https://arxiv.org/pdf/2408.15793) · [DOI 10.48550/arxiv.2408.15793](https://doi.org/10.48550/arxiv.2408.15793) · [arXiv 2408.15793](https://arxiv.org/abs/2408.15793) · [OpenAlex](https://openalex.org/W4402706056) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary The paper asks whether an academic lab with only a handful of server-grade GPUs and a few days of access can usefully adapt a 7B LLM to a new language through continued pretraining. Starting from the LeoLM recipe for German, the authors shrink it to fit such a budget and adapt Mistral-7B to German, using filtered OSCAR23.01 data. They later run a follow-up "hindsight" study on Arabic. Two efficiency levers are tested. The first is training entirely in bfloat16 instead of mixed precision. The second is replacing the original tokenizer with a language-specific one. Pure bfloat16 proves fast and essentially harmless for continued pretraining. Tokenizer swapping gives more compact tokenization and matches the original vocabulary in quality, but it does not raise German scores. The German models actually fall below the untouched base model, while the Arabic models beat several baselines. The practical lesson is that adaptation pays off mainly for languages the base model covers poorly. It also shows that cheap training tricks can make such adaptation feasible on small budgets. ## Key points - The budget is 7,680 steps with batch size 256 and 4,096-token context, about 8 billion tokens in total. That is 12.5% of LeoLM's 64 billion tokens. The learning rate is raised to 4 × 10⁻⁵, and the data is German only, with no English or code mixed in. - On two 80GB GPUs, pure bfloat16 ran 39% faster than mixed precision. On a single 80GB GPU, mixed precision could not run at all because of out-of-memory errors. - In bfloat16, an update u is lost when u \< W/128, since there are only 7 mantissa bits. Mistral-7B's RMSNorm weights are much larger than its other weights, so they effectively stayed frozen. This had no visible harm on loss or downstream tasks. - The swapped tokenizer is a sentencepiece BPE model with byte-fallback and a vocabulary of 32,768. That size was chosen to suit Nvidia Tensor Cores. The tokenizer was trained on a subset of the German training data. - German adaptation hurt performance. For comparison, LeoLM spent 8× the compute and gained only 0.6 points. Arabic, whose script the original tokenizer did not cover, improved significantly with tokenizer swapping. ## Interesting ideas - Before choosing a low-precision regime, plot weight magnitudes per parameter type and compare them with typical update sizes; this reveals which parameters will silently stop learning. - Replacing a vocabulary, rather than extending it, keeps the embedding and softmax matrices small, and sizing the vocabulary to a hardware-friendly number speeds up training. - Choose a target that the base model already handles well as a deliberate lower bound, then confirm with a second, poorly covered target; this separates the effect of the method from the effect of how much room there is to improve. - Measure tokenizer quality by fertility, meaning how many words a fixed compute budget covers, not by token counts alone. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): When training gparse in bf16, check norm and bias parameters against learning-rate-scaled updates so frozen layers are found by analysis, not by accident. - **I2** (tree): The subword vocabulary of gparse's transformer encoder could be retrained on English treebank-style text at a Tensor-Core-friendly size, giving cheaper training and more sensible word pieces. - **I3** (learn): Silver-data retraining experiments should report gains both on domains where the parser is already strong and on weak ones, with the untouched model as a negative control. - **I4** (tokens): Comparing the crate en tokenizer with gparse's subword splitter in words per token would show where segmentation wastes compute or breaks morphology. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works 🧭 Topics (2): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works · [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) · 124 works ## Cite Cite as: Konstantin Dobler and Gerard de Melo. 2024. Language Adaptation on a Tight Academic Compute Budget: Tokenizer Swapping Works and Pure bfloat16 Is Enough. arXiv:2408.15793. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{dobler2024language, title = {{Language Adaptation on a Tight Academic Compute Budget: Tokenizer Swapping Works and Pure bfloat16 Is Enough}}, author = {Konstantin Dobler and Gerard {de Melo}}, year = {2024}, doi = {10.48550/arxiv.2408.15793}, eprint = {2408.15793}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2408.15793} } ``` - [← previous Jailbreak Attack for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/li-2024-jailbreak-attack-large/) - [next → Large Language Models and Biblical Hebrew: Limitations, pitfalls, opportunities](https://md.research.elamur.ai/free/en/papers/staps-2024-large-language-models-biblical-hebrew/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/16/) ## References 2 references, 2 of them with a card of ours. ↗ leads to the work. 1. Abadji, J., Ortiz Suarez, P., Romary, L., and Sagot, B. [Towards a cleaner document-oriented multilingual crawled corpus](https://md.research.elamur.ai/free/en/papers/abadji-2022-towards-cleaner-document-oriented-multilingual/). In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 4344–4355. European Language Resources Association, 2022. URL https://aclanthology.org/2022.lrec-1.463. [↗](https://aclanthology.org/2022.lrec-1.463/) 2. Almazrouei, E., Cojocaru, R.-A., Baldo, M., Malartic, Q., Alobeidli, H., Mazzotta, D., Penedo, G., Campesan, G., Farooq, M., Alhammadi, M., Launay, J., and Noune, B. [AlGhafa evaluation benchmark for Arabic language models](https://md.research.elamur.ai/free/en/papers/almazrouei-2023-alghafa-evaluation-benchmark-arabic-language/). In Sawaf, H., El-Beltagy, S. R., Zaghouani, W., Magdy, W., Abdelali, A., Tomeh, N., Farha, I. A., Habash, N., Khalifa, S., Keleg, A., Haddad, H., Zitouni, I., Mrini, K., and Al-Matham, R. N. (eds.), ARABICNLP, pp. 244– 275. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.arabicnlp-1.21. URL https://api.semanticscholar.org/CorpusID:265607930. [↗](https://doi.org/10.18653/v1/2023.arabicnlp-1.21) *Updated 2 November 2026*