--- title: "How to age BERT Well — paper by Harju and Goot (2025)" description: "This paper asks whether continuing to pre-train an existing BERT on raw text from an older stage of a… 5 references, cited by 3 papers here." authors: ["Anika Harju", "Rob van der Goot"] year: 2025 venue: "Proceedings of the First Workshop on Language Models for Low-Resource Languages" source: "https://aclanthology.org/2025.loreslm-1.21/" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/harju-2025-how-age-bert-well-continuous/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/harju-2025-how-age-bert-well-continuous/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/13/) / How to age BERT Well: Continuous Training for Historical Language Adaptation # How to age BERT Well: Continuous Training for Historical Language Adaptation **Authors:** Anika Harju and [Rob van der Goot](https://md.research.elamur.ai/free/en/authors/rob-van-der-goot/) · **Year:** 2025 · **Venue:** [Proceedings of the First Workshop on Language Models for Low-Resource Languages](https://md.research.elamur.ai/free/en/venues/loreslm/) ([Workshop on Language Models for Low-Resource Languages 2025](https://md.research.elamur.ai/free/en/venues/loreslm/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.loreslm-1.21/) · [PDF](https://aclanthology.org/2025.loreslm-1.21.pdf) · [OpenAlex](https://openalex.org/W7080350460) **Citations:** 0 citations (OpenAlex, 2026-10-14) · 3 in our library, of the 149,439 papers whose bibliographies we read ## Summary Taggers and parsers for historical languages struggle because these varieties have little annotated data, unstable spelling and rich inflection. This paper asks whether continuing to pre-train an existing BERT on raw text from an older stage of a language helps. The authors take Old English (OE) and continue masked language modeling for two starting models, English bert-base-uncased and bert-base-multilingual-uncased. They use roughly two million words of unlabeled OE from the Complete Corpus of Anglo-Saxon Poetry and a 2018 Plaintext Wikipedia dump. They then fine-tune both models for POS tagging on the manually annotated ISWOC corpus, which is written in West Saxon. Continued pre-training improves tagging for both models, both in-domain and on held-out texts. The two models rank differently depending on the measure. The English model does better at masked-token prediction, but the multilingual model tags better and transfers better to unseen texts. The result shows that a cheap, label-free adaptation step can substantially help low-resource historical varieties. It also warns that language-modeling scores are a poor guide for choosing the best base model. ## Key points - Data and splits: - The unlabeled OE pre-training text totals about 2.04M words; fully capitalized texts were removed. - ISWOC provides 2,541 annotated sentences (29,406 tokens). - Of these, 2,230 sentences are used for train/dev, and 311 sentences from Ælfric's Lives of Saints and Orosius form an out-of-domain test set. - In-domain POS accuracy, before → after continued pre-training: - EN BERT: 86.37 → 92.16. - mBERT: 88.79 → 93.70. - Out-of-domain POS accuracy, before → after: - EN BERT: 71.96 → 76.71. - mBERT: 77.87 → 84.13. - The adaptation gain is larger on unseen texts. - Best hyperparameters: - Masked language modeling: peak learning rate 1e-3, batch size 16. - POS fine-tuning: learning rate 2e-5, batch size 8. - EN BERT reached about 83% masked-token accuracy. - Per-tag results across 18 OE tags: - Personal pronouns (Pp) score best, with F1 0.98 for mBERT and 0.97 for EN BERT. - Both models never correctly tag foreign words (F-) or interjections (I-). - EN BERT never predicts interrogative adverbs (Du), even though they occur 53 times in the Gospels. ## Interesting ideas - Running extra masked-LM training on unlabeled target-variety text before supervised fine-tuning is a cheap adaptation step, and its benefit is largest out of domain. - The model that wins on masked-token prediction lost on the downstream task, so intrinsic LM scores should not be used to pick a base model or checkpoint. - Before training, the authors checked how each tokenizer handles OE-specific letters and found that one splits them into isolated characters while the other only knows them inside other languages' subwords. - The models were sometimes highly confident in wrong tags and unsure about correct ones, so their softmax scores are a poor filter for correctness. - A per-tag breakdown exposed rare categories that a model never outputs at all, which overall accuracy completely hides. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (tree): gparse's encoder could get a round of continued pre-training on raw fables, tales and school texts before parser fine-tuning, to make it more robust on those genres. - **I2** (learn): select encoder checkpoints by downstream parse/tag accuracy on held-out treebank data, never by LM loss alone. - **I3** (tokens): routinely audit how the en crate and gparse's subword vocabulary segment rare characters, archaic spellings or symbols, since bad segmentation silently caps tagger quality. - **I4** (learn): when building silver data, rely on agreement between several witness parsers rather than one model's confidence to accept or reject a tree. - **I5** (tokens): add an evaluation report that flags any UPOS/XPOS tag with zero predictions or near-zero recall, so rare-class gaps get targeted training data. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [IT University of Copenhagen](https://md.research.elamur.ai/free/en/universities/it-university-of-copenhagen/) — University, Copenhagen, Denmark · 27 works · 2,406 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works 🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works ## Cite Cite as: Anika Harju and Rob van der Goot. 2025. How to age BERT Well: Continuous Training for Historical Language Adaptation. Proceedings of the First Workshop on Language Models for Low-Resource Languages. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{harju2025how, title = {{How to age BERT Well: Continuous Training for Historical Language Adaptation}}, author = {Anika Harju and Rob {van der Goot}}, booktitle = {Proceedings of the First Workshop on Language Models for Low-Resource Languages}, year = {2025}, url = {https://aclanthology.org/2025.loreslm-1.21/} } ``` - [← previous Graders Should Cheat: Privileged Information Enables Expert-Level Automated Evaluations](https://md.research.elamur.ai/free/en/papers/zhou-2025-graders-should-cheat-privileged-information/) - [next → ImpactAi at ImageEval 2025 Shared Task: Region-Aware Transformers for Arabic Image Captioning – A Case Study on the Palestinian Narrative](https://md.research.elamur.ai/free/en/papers/qasem-2025-impactai-imageeval-2025-shared-task/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/13/) ## References 5 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Peter S Baker. 2012. Introduction to old English. John Wiley & Sons. [find](https://scholar.google.com/scholar?q=Peter+S+Baker%2E+2012%2E+Introduction+to+old+English%2E+John+Wiley+%26+Sons%2E) 2. Alistair Baron and Paul Rayson. 2008. Vard2: A tool for dealing with spelling variation in historical corpora. [find](https://scholar.google.com/scholar?q=Alistair+Baron+and+Paul+Rayson%2E+2008%2E+Vard2%3A+A+tool+for+dealing+with+spelling+variation+in+historical+corpora%2E) 3. Christin Beck, Hannah Booth, Mennatallah El-Assady, and Miriam Butt. 2020. Representation problems in linguistic annotations: Ambiguity, variation, uncertainty, error and bias. In Proceedings of the 14th Linguistic Annotation Workshop, pages 60–73, Barcelona, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Christin+Beck%2C+Hannah+Booth%2C+Mennatallah+El%2DAssady%2C+and+Miriam+Butt%2E+2020%2E+Representation+problems+in+linguistic+annotations%3A+Ambiguity%2C+variation%2C+uncertainty%2C+error+and+bias%2E+In+Proceedings+of+the+1) 4. Taylor Berg-Kirkpatrick, Alexandre Bouchard-Côté, John DeNero, and Dan Klein. 2010. [Painless unsupervised learning with features](https://md.research.elamur.ai/free/en/papers/kirkpatrick-2010-painless-unsupervised-learning-features/). In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 582–590. 5. Erica Biagetti, Martina Giuliani, Silvia Zampetta, Silvia Luraghi, and Chiara Zanchi. 2024. Combining neo-structuralist and cognitive approaches to semantics to build wordnets for ancient languages: Challenges and perspectives. In Proceedings of the Workshop on Cognitive Aspects of the Lexicon @ LREC-COLING 2024, pages 151–161, Torino, Italia. ELRA and ICCL. [find](https://scholar.google.com/scholar?q=Erica+Biagetti%2C+Martina+Giuliani%2C+Silvia+Zampetta%2C+Silvia+Luraghi%2C+and+Chiara+Zanchi%2E+2024%2E+Combining+neo%2Dstructuralist+and+cognitive+approaches+to+semantics+to+build+wordnets+for+ancient+languages%3A+C) *Updated 31 October 2026*