---
title: "Continual Learning Under Language Shift — Gogoulou 2023"
description: "The authors train GPT-style models with 126M, 356M and 1.3B parameters, first on English. 4 references, cited by 16 papers here."
authors: ["Evangelia Gogoulou", "Timothee Lesort", "Magnus Boman", "Joakim Nivre"]
year: 2023
venue: "arXiv:2311.01200"
source: "https://arxiv.org/abs/2311.01200"
doi: "10.48550/arxiv.2311.01200"
arxiv: "2311.01200"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gogoulou-2023-continual-learning-under-language-shift/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gogoulou-2023-continual-learning-under-language-shift/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/) / Continual Learning Under Language Shift

# Continual Learning Under Language Shift

**Authors:** Evangelia Gogoulou, Timothee Lesort, Magnus Boman and [Joakim Nivre](https://md.research.elamur.ai/free/en/authors/joakim-nivre/) · **Year:** 2023 · **Venue:** [arXiv:2311.01200](https://md.research.elamur.ai/free/en/venues/lecture-notes-in-computer-science/) ([Lecture notes in computer science 2023](https://md.research.elamur.ai/free/en/venues/lecture-notes-in-computer-science/2023/)) · **Publisher:** [Springer Science+Business Media](https://md.research.elamur.ai/free/en/publishers/springer-science-business-media/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2311.01200) · [PDF](https://arxiv.org/pdf/2311.01200) · [DOI 10.48550/arxiv.2311.01200](https://doi.org/10.48550/arxiv.2311.01200) · [arXiv 2311.01200](https://arxiv.org/abs/2311.01200) · [OpenAlex](https://openalex.org/W4402081998)

**Citations:** 8 citations (OpenAlex, 2026-10-14) · [16](https://md.research.elamur.ai/free/en/papers/gogoulou-2023-continual-learning-under-language-shift/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Retraining large language models from scratch each time new data appears is costly. This paper asks what happens when an existing model is instead updated with text in previously unseen languages. The authors train GPT-style models with 126M, 356M and 1.3B parameters, first on English. They then add Danish, Icelandic and Norwegian one at a time, producing 15 English-first sequences of two to four languages. Each language gets an equal amount of training data from the Nordic Pile, and all models share the GPT-SW3 tokenizer. The authors measure transfer to new languages (forward) and to earlier ones (backward) using test cross-entropy, and compare against monolingual and jointly trained baselines. Gains on newly added languages are consistently positive regardless of order. The effect on earlier languages varies: Icelandic is linked to strong forgetting, while Danish and Norwegian interact more mildly and can even benefit English. Bigger models reach lower loss but show the same transfer patterns. Accidental language contamination combined with syntactic similarity explains the results best, which can guide continual multilingual pre-training.

## Key points
- **Training setup:** Each continual stage runs 35K steps (roughly 36M sequences of 2048 tokens). Models use the 64K-token BPE GPT-SW3 tokenizer and are implemented in Nvidia NeMo. The joint en+da+is+no baseline is trained for 105K steps.
- **Data sources:** The Pile, Wikipedia, mc4, OSCAR, Icelandic Gigaword, Danish Gigaword and NCC.
- **TDS metric:** The authors propose token distribution similarity (TDS), the cosine similarity between token-frequency vectors of two corpora. It ranks da-no highest (0.92) and en-is lowest (0.35). However, it fails to predict the weak Danish–Norwegian backward transfer. Among the URIEL features, only syntactic similarity (SYN) fits the observed patterns.
- **Contamination analysis:** fastText language identification shows the Icelandic corpus is almost purely Icelandic (99.32%), and other corpora contain negligible Icelandic. The Norwegian and Danish corpora include 3.16% and 1.32% English text, respectively.
- **Effect of scale:** Compared with 126M, final average test loss drops by about 8% at 356M and 11% at 1.3B. Forgetting of earlier languages stays largely unchanged.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Université de Montréal](https://md.research.elamur.ai/free/en/universities/universite-de-montreal/) — University, Montreal, Canada · 68 works · 79,382 citations
- [Uppsala University](https://md.research.elamur.ai/free/en/universities/uppsala-university/) — University, Uppsala, Sweden · 64 works · 13,164 citations
- [Mila - Quebec Artificial Intelligence Institute](https://md.research.elamur.ai/free/en/labs/mila-quebec-artificial-intelligence-institute/) — Research institute, Montreal, Canada · 54 works · 13,565 citations
- [KTH Royal Institute of Technology](https://md.research.elamur.ai/free/en/universities/kth-royal-institute-of-technology/) — University, Stockholm, Sweden · 18 works · 788 citations
- [Karolinska Institutet](https://md.research.elamur.ai/free/en/universities/karolinska-institutet/) — University, Stockholm, Sweden · 3 works · 134 citations
- [RISE Research Institutes of Sweden](https://md.research.elamur.ai/free/en/labs/rise-research-institutes-of-sweden/) — Research institute, Gothenburg, Sweden · 2 works · 33 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

## Cite

Cite as: Evangelia Gogoulou, Timothee Lesort, Magnus Boman and Joakim Nivre. 2023. Continual Learning Under Language Shift. arXiv:2311.01200.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{gogoulou2023continual,
  title = {{Continual Learning Under Language Shift}},
  author = {Evangelia Gogoulou, Timothee Lesort, Magnus Boman and Joakim Nivre},
  year = {2023},
  doi = {10.48550/arxiv.2311.01200},
  eprint = {2311.01200},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.01200}
}
```

- [← previous Arabic Fine-Grained Entity Recognition](https://md.research.elamur.ai/free/en/papers/liqreina-2023-arabic-fine-grained-entity-recognition/)
- [next → From Zero to Hero: Examining the Power of Symbolic Tasks in Instruction Tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-zero-hero-examining-power-symbolic/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/)

## References

4 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 1. Abnar, S., Dehghani, M., Neyshabur, B., Sedghi, H.: Exploring the limits of large scale pre-training. In: International Conference on Learning Representations (2021) 2. Aljundi, R., Belilovsky, E., Tuytelaars, T., Charlin, L., Caccia, M., Lin, M., Page-Caccia, L.: Online continual learning with m… [find](https://scholar.google.com/scholar?q=1%2E+Abnar%2C+S%2E%2C+Dehghani%2C+M%2E%2C+Neyshabur%2C+B%2E%2C+Sedghi%2C+H%2E%3A+Exploring+the+limits+of+large+scale+pre%2Dtraining%2E+In%3A+International+Conference+on+Learning+Representations+%282021%29+2%2E+Aljundi%2C+R%2E%2C+Belilovsky%2C+E%2E%2C)
2. Continual Learning Under Language Shift 8. Coria, J.M., Veron, M., Ghannay, S., Bernard, G., Bredin, H., Galibert, O., Rosset, S.: Analyzing BERT cross-lingual transfer capabilities in continual sequence labeling. In: Proceedings of the First Workshop on Performance and Interpretability Evaluations… [find](https://scholar.google.com/scholar?q=Continual+Learning+Under+Language+Shift+8%2E+Coria%2C+J%2EM%2E%2C+Veron%2C+M%2E%2C+Ghannay%2C+S%2E%2C+Bernard%2C+G%2E%2C+Bredin%2C+H%2E%2C+Galibert%2C+O%2E%2C+Rosset%2C+S%2E%3A+Analyzing+BERT+cross%2Dlingual+transfer+capabilities+in+continual+seque)
3. Wang, J., Zettlemoyer, L., Kozareva, Z., Diab, M., Stoyanov, V., Li, X.: Few-shot learning with multilingual generative language models. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pp. 9019–9052 (2022) 25. Littell, P., Mortensen, D.R., Lin, K., Kairis,… [find](https://scholar.google.com/scholar?q=Wang%2C+J%2E%2C+Zettlemoyer%2C+L%2E%2C+Kozareva%2C+Z%2E%2C+Diab%2C+M%2E%2C+Stoyanov%2C+V%2E%2C+Li%2C+X%2E%3A+Few%2Dshot+learning+with+multilingual+generative+language+models%2E+In%3A+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+)
4. OpenAI (2018) 34. Ramasesh, V.V., Lewkowycz, A., Dyer, E.: Effect of scale on catastrophic forgetting in neural networks. In: International Conference on Learning Representations (2022) 35. Scao, T.L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A.S., Yvon, F., G… [find](https://scholar.google.com/scholar?q=OpenAI+%282018%29+34%2E+Ramasesh%2C+V%2EV%2E%2C+Lewkowycz%2C+A%2E%2C+Dyer%2C+E%2E%3A+Effect+of+scale+on+catastrophic+forgetting+in+neural+networks%2E+In%3A+International+Conference+on+Learning+Representations+%282022%29+35%2E+Scao%2C+T%2EL%2E)

*Updated 30 October 2026*