--- title: "Adapting Multilingual Embedding Models to Historical" description: "Off-the-shelf multilingual sentence embeddings struggle with digitized historical Luxembourgish, a… 3 references, cited by 3 papers here." authors: ["Andrianos Michail", "Corina Raclé", "Juri Opitz", "Simon Clematide"] year: 2025 venue: "Proceedings of the 9th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2025)" source: "https://aclanthology.org/2025.latechclfl-1.26/" doi: "10.18653/v1/2025.latechclfl-1.26" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/michail-2025-adapting-multilingual-embedding-models-historical/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/michail-2025-adapting-multilingual-embedding-models-historical/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/) / Adapting Multilingual Embedding Models to Historical Luxembourgish # Adapting Multilingual Embedding Models to Historical Luxembourgish **Authors:** [Andrianos Michail](https://md.research.elamur.ai/free/en/authors/andrianos-michail/), Corina Raclé, [Juri Opitz](https://md.research.elamur.ai/free/en/authors/juri-opitz/) and [Simon Clematide](https://md.research.elamur.ai/free/en/authors/simon-clematide/) · **Year:** 2025 · **Venue:** [Proceedings of the 9th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2025)](https://md.research.elamur.ai/free/en/venues/latechclfl/) ([LaTeCH-CLfL 2025](https://md.research.elamur.ai/free/en/venues/latechclfl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.latechclfl-1.26/) · [PDF](https://aclanthology.org/2025.latechclfl-1.26.pdf) · [DOI 10.18653/v1/2025.latechclfl-1.26](https://doi.org/10.18653/v1/2025.latechclfl-1.26) · [OpenAlex](https://openalex.org/W4411120107) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/michail-2025-adapting-multilingual-embedding-models-historical/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Off-the-shelf multilingual sentence embeddings struggle with digitized historical Luxembourgish, a low-resource language. These newspaper texts contain OCR errors and spelling that predates standardization. The authors sampled roughly 2,340 Luxembourgish articles from the BNL newspaper archive (1841–1948), choosing them so that many topics are represented. They then had GPT-4o split each article into sentences and translate it into modern German, French and English, which produced about 20,000 parallel sentences per language pair. From held-out articles they built a cross-lingual retrieval benchmark called Historical LB Bitext Mining. They adapted M-MPNet, LaBSE, M-GTE and LuxEmbedder, using either contrastive training or knowledge distillation. Every model became more accurate on historical retrieval, with gains of up to 13pp. However, scores on modern Luxembourgish tasks, especially adversarial paraphrase detection, tended to fall. Balancing historical pairs one-to-one with clean modern pairs largely prevented that loss. The best all-round model is an M-GTE trained on this mix. The models and bitexts are publicly released, which helps archives offer semantic search across languages. ## Key points - **Data sampling:** Articles were clustered into 2,000 groups with K-Means applied to the output of a 100-topic LDA model. Only clusters containing over 20 articles were retained, 605 in total. From each, the authors took the most representative article plus three random ones, each 5–20 sentences long. - **Translation and quality checks:** - GPT-4o yielded about 22,500 sentence pairs per language pair, and 65.0% of sentences formed exact four-way parallel sets. - In 1.4% of cases the Luxembourgish sentence GPT-4o reproduced did not match the source exactly, and these were fixed by hand. - A native speaker checked 100 sentence quadruplets. 88 were comprehensible or confidently guessable, and 88.6% of their translations were rated adequate in each target language. - **Test set:** 233 articles (LB-DE 2,127, LB-FR 2,157, LB-EN 2,105 sentences). The score is bidirectional accuracy against about 2k candidates. Candidates with Levenshtein similarity above 0.85 to the source were removed. - **Training data:** - Historical: LB-DE 20,092, LB-FR 20,010 and LB-EN 19,054 pairs. - Modern: 89,405 LB-FR and 28,172 LB-EN pairs from RTL.lu, taken from earlier LuxEmbedder work. - Modern-language evaluation used ParaLux (312 triplets) and SIB 200 (LB). - **Final model:** M-GTE trained on 120,000 pairs, half historical and half modern. It averages 97.5% on historical bitext mining. It beats the adapted LaBSE and LuxEmbedder by about 6pp on SIB-200 and about 20pp on ParaLux. ## Interesting ideas - After an LLM rewrites a source sentence (here, segmenting it and repairing OCR errors), compare the regenerated source to the original string exactly and hand-correct only the mismatches; this cheaply catches the model drifting from its input. - Choose which documents to annotate by clustering topic-model output and taking one central plus a few random examples from each sizeable cluster; this spreads a small annotation budget over many themes. - When building a retrieval test, drop candidates that are almost identical in surface form to the query, so that legitimate near-duplicates are not scored as errors. - Pair every noisy in-domain training example with a clean general-domain one at a 1:1 ratio to limit regression on the original domain. - Include an adversarial test where the negative example looks lexically almost identical to the positive but differs in meaning (as in ParaLux); it exposes losses of fine semantic sensitivity that easier benchmarks hide. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): Apply the same exact-match check whenever an LLM teacher returns tokenized or segmented text alongside its labels, so any silent rewriting is detected before the output enters silver data. - **I2** (learn): Use this procedure to select fables or school texts for LLM-teacher labelling, so the silver corpus is not dominated by a few frequent story types. - **I3** (answer): In QA or test-item matching, filter out distractor answers that are near-copies of the gold answer, so measured accuracy reflects real understanding rather than artefacts in the data. - **I4** (tree): When retraining the perceptron parser or gparse on noisy or dialect-converted treebanks, mix in an equal amount of standard UD data and track LAS on the clean set to see whether the parser regresses. - **I5** (learn): Add minimal-pair negative controls, such as swapped roles or negation, to frame evaluation so that a repair rule or retraining that blurs who-did-what distinctions becomes visible. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Zurich](https://md.research.elamur.ai/free/en/universities/university-of-zurich/) — University, Zurich, Switzerland · 78 works · 12,260 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [📚 Literature and Literary Theory](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/literature-and-literary-theory/) · 519 works 🧭 Topics (1): [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works ## Cite Cite as: Andrianos Michail, Corina Raclé, Juri Opitz and Simon Clematide. 2025. Adapting Multilingual Embedding Models to Historical Luxembourgish. Proceedings of the 9th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2025). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{michail2025adapting, title = {{Adapting Multilingual Embedding Models to Historical Luxembourgish}}, author = {Andrianos Michail, Corina Raclé, Juri Opitz and Simon Clematide}, booktitle = {Proceedings of the 9th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2025)}, year = {2025}, doi = {10.18653/v1/2025.latechclfl-1.26}, url = {https://doi.org/10.18653/v1/2025.latechclfl-1.26} } ``` - [← previous A Survey of QUD Models for Discourse Processing](https://md.research.elamur.ai/free/en/papers/fu-2025-survey-qud-models-discourse-processing/) - [next → Adapting Psycholinguistic Research for LLMs: Gender-inclusive Language in a Coreference Context](https://md.research.elamur.ai/free/en/papers/bartl-2025-adapting-psycholinguistic-research-llms-gender/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/) ## References 3 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Cedric Lothritz, Bertrand Lebichot, Kevin Allix, Lisa Veiber, Tegawende Bissyande, Jacques Klein, Andrey Boytsov, Clément Lefebvre, and Anne Goujon. 2022. LuxemBERT: Simple and practical data augmentation in language model pre-training for Luxembourgish. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5080–5089, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Cedric+Lothritz%2C+Bertrand+Lebichot%2C+Kevin+Allix%2C+Lisa+Veiber%2C+Tegawende+Bissyande%2C+Jacques+Klein%2C+Andrey+Boytsov%2C+Cl%C3%A9ment+Lefebvre%2C+and+Anne+Goujon%2E+2022%2E+LuxemBERT%3A+Simple+and+practical+data+augmenta) 2. David Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba Alabi, Yanke Mao, Haonan Gao, and En-Shiun Lee. 2024. [SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects](https://md.research.elamur.ai/free/en/papers/adelani-2023-sib-200-simple-inclusive-big/). In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Andrianos Michail, Simon Clematide, and Juri Opitz. 2025. PARAPHRASUS: A comprehensive benchmark for evaluating paraphrase detection models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 8749–8762, Abu Dhabi, UAE. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2309.07445) 3. Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. [MTEB: Massive text embedding benchmark](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-mteb-massive-text-embedding-benchmark/). In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2023.eacl-main.148) *Updated 1 November 2026*