---
title: "Language corpora for the Dutch medical domain — Es 2026"
description: "Dutch has very little openly available medical text. 24 references. Among them: 0."
authors: ["B. van Es"]
year: 2026
venue: "arXiv:2604.25374"
source: "https://arxiv.org/abs/2604.25374"
doi: "10.48550/arxiv.2604.25374"
arxiv: "2604.25374"
licence: "CC BY-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/es-2026-language-corpora-dutch-medical-domain/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/es-2026-language-corpora-dutch-medical-domain/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/34/) / Language corpora for the Dutch medical domain

# Language corpora for the Dutch medical domain

**Authors:** B. van Es · **Year:** 2026 · **Venue:** arXiv:2604.25374 · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2604.25374) · [PDF](https://arxiv.org/pdf/2604.25374) · [DOI 10.48550/arxiv.2604.25374](https://doi.org/10.48550/arxiv.2604.25374) · [arXiv 2604.25374](https://arxiv.org/abs/2604.25374) · [OpenAlex](https://openalex.org/W7157918723)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Dutch has very little openly available medical text. General Dutch corpora contain few medical documents, and clinical records are locked behind privacy rules. This makes it hard to pre-train medical language models for Dutch. The paper assembles a large open Dutch medical corpus using three complementary routes:
- **Machine translation** of English biomedical collections such as PubMed, PMC, MIMIC and Apollo, using NMT systems and several commercial LLMs.
- **Classification** of Dutch web and PDF crawls to pick out medical documents.
- **Harvesting** of open Dutch sources, such as rights-free PhD theses obtained through OAI, medical journal articles and clinical guidelines.

The texts are cleaned and, at the combined level, de-identified. The headline figures are roughly 35–36 billion tokens. The document count is reported as 73.5M in the table and about 105M in the abstract. Everything is released on Hugging Face, together with a translated question-answering set for instruction tuning and several Dutch medical encoder and decoder models. The work matters because it gives Dutch clinical NLP its first large pre-training resource. The pipeline is reusable for other mid-resource languages, though translated text remains an approximation of the originals.

## Key points
- The biggest components are FineWeb2 medical selections (11.5B words, 11.7M docs), PMC OA non-commercial (8.7B words) and PMC OA commercial (4.4B words). MIMIC III/IV translations add 2.9B words and PubMed abstracts 2.6B words.
- The medical/non-medical filter was built in two steps. First, GPT4.1-nano labelled 100,000 random OSCAR samples. Then a dense layer trained on a frozen RobBERT2023 encoder was applied to FineWeb2 and FinePDFs.
- Translation tools included NLLB, MariaNMT, GPT3.5/4o, and Gemini 1.5/2/2.5. MedQA and BioASQ were translated with GPT4o-mini and MIRIAD with GPT4.1-nano. PMC patient cases were rewritten into discharge-letter format.
- The combined set DutchMedicalTextV3 was de-identified with the rule-based DEDUCE tool. The individual sub-datasets were not.
- Models trained on the data include CardioLlama.nl and MedLlama.nl (Llama 3.2, 1B), CardioBERTa.nl (continued from MedRoBERTa.nl, 120M) and CardioDeBERTa.nl (DeBERTaV2 trained from scratch, 400M). Tooling lives in the PubScience library.

## Interesting ideas
- Use an expensive LLM to label a modest random sample, then train a cheap probe on a frozen encoder to filter a huge corpus; this scales domain selection at low cost.
- Split long texts into non-overlapping chunks cut at sentence boundaries, each sized to about half the model context so the other half is left for the output.
- Detect scanned PDFs with a layered heuristic: check producer metadata first, then the share of pages carrying extractable text, then send failures to OCR through a fallback chain of parsers.
- Filter Dutch/English summary pairs from theses by checking cross-lingual sentence-embedding similarity before admitting them as parallel data.
- Rewrite source documents into another genre, such as case reports into structured discharge letters, to create in-domain text of a format that is otherwise unavailable.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Have the LLM teacher tier tag a few thousand sentences by domain or difficulty, then train a light classifier over Mova's own features to choose which raw text enters silver-data parsing.
- **I2** (back): When the compressor or meta-world translation regenerates long documents, cut input at sentence boundaries and budget output length per chunk, so generation never silently truncates.
- **I3** (tokens): Build ingestion as an explicit, logged fallback chain with a stated reason for each switch, so every downstream token can be traced back to how its text was obtained.
- **I4** (back): A similarity score can serve as a cheap first-pass negative control on text→graph→text round-trips, flagging gross failures before the slower, explainable graph-level comparison.
- **I5** (answer): Turn fables and tales into school-test formats, such as question sheets or fill-in exercises, to grow evaluation material whose answers are grounded in a known source text.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

Also on the byline: [University Medical Center Utrecht](https://ror.org/0575yy874).

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🧬 Biochemistry, Genetics and Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/) · 234 works · [🧬 Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/molecular-biology/) · 231 works

## Cite

Cite as: B. van Es. 2026. Language corpora for the Dutch medical domain. arXiv:2604.25374.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{es2026language,
  title = {{Language corpora for the Dutch medical domain}},
  author = {B. {van Es}},
  year = {2026},
  doi = {10.48550/arxiv.2604.25374},
  eprint = {2604.25374},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2604.25374}
}
```

- [← previous Language Acquisition Device in Large Language Models](https://md.research.elamur.ai/free/en/papers/mita-2026-language-acquisition-device-large-language/)
- [next → Language Effects in Text-to-SQL Across English and Portuguese](https://md.research.elamur.ai/free/en/papers/nobre-2026-language-effects-text-sql-across/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/34/)

## References

24 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, et al. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). In: Journal of Machine Learning Research. vol. 21; 2020. p. 1–67.
2. Gao L, Biderman S, Black S, Golding L, Hoppe T, Foster C, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:210100027. 2020;. [find](https://scholar.google.com/scholar?q=Gao+L%2C+Biderman+S%2C+Black+S%2C+Golding+L%2C+Hoppe+T%2C+Foster+C%2C+et+al%2E+The+Pile%3A+An+800GB+dataset+of+diverse+text+for+language+modeling%2E+arXiv+preprint+arXiv%3A210100027%2E+2020%3B%2E)
3. Gu Y, Tinn R, Cheng H, Lucas M, Usuyama N, Liu X, et al. [Domain-specific language model pretraining for biomedical natural language processing](https://md.research.elamur.ai/free/en/papers/gu-2020-domain-specific-language-model-pretraining/). ACM Transactions on Computing for Healthcare (HEALTH). 2021;3(1):1–23.
4. Névéol A, Dalianis H, Velupillai S, Savova G, Zweigenbaum P. Clinical natural language processing in languages other than English: opportunities and challenges. Journal of biomedical semantics. 2018;9(1):12. [find](https://scholar.google.com/scholar?q=Ne%CC%81ve%CC%81ol+A%2C+Dalianis+H%2C+Velupillai+S%2C+Savova+G%2C+Zweigenbaum+P%2E+Clinical+natural+language+processing+in+languages+other+than+English%3A+opportunities+and+challenges%2E+Journal+of+biomedical+semantics%2E+2018)
5. Schäfer R. Processing and querying large web corpora with the COW14 architecture. In: Proceedings of the 3rd Workshop on Challenges in the Management of Large Corpora; 2015. p. 28–34. [find](https://scholar.google.com/scholar?q=Scha%CC%88fer+R%2E+Processing+and+querying+large+web+corpora+with+the+COW14+architecture%2E+In%3A+Proceedings+of+the+3rd+Workshop+on+Challenges+in+the+Management+of+Large+Corpora%3B+2015%2E+p%2E+28%E2%80%9334%2E)
6. Ortiz Suárez PJ, Romary L, Sagot B. [A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages](https://md.research.elamur.ai/free/en/papers/suarez-2020-monolingual-approach-contextualized-word-embeddings/). In: Jurafsky D, Chai J, Schluter N, Tetreault J, editors. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics; 2020. p. 1703–1714. Available from: https://aclanthology.org/2020.acl-main.156/. [↗](https://aclanthology.org/2020.acl-main.156/)
7. Ordelman R, de Jong F, Van Hessen A, Hondorp H. TWnC: a multifaceted Dutch news corpus. ELRA Newsletter. 2007;12(3/4):4–7. [find](https://scholar.google.com/scholar?q=Ordelman+R%2C+de+Jong+F%2C+Van+Hessen+A%2C+Hondorp+H%2E+TWnC%3A+a+multifaceted+Dutch+news+corpus%2E+ELRA+Newsletter%2E+2007%3B12%283%2F4%29%3A4%E2%80%937%2E)
8. Oostdijk N, Reynaert M, Monachesi P, Van Noord G, Ordelman R, Schuurman I, et al. From D-Coi to SoNaR: a reference corpus for Dutch. In: Calzolari N, Choukri K, Maegaard B, Mariani J, Odijk J, Piperidis S, et al., editors. Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08). Marrakech, Morocco: European Language Resources Association (ELRA); 2008. [find](https://scholar.google.com/scholar?q=Oostdijk+N%2C+Reynaert+M%2C+Monachesi+P%2C+Van+Noord+G%2C+Ordelman+R%2C+Schuurman+I%2C+et+al%2E+From+D%2DCoi+to+SoNaR%3A+a+reference+corpus+for+Dutch%2E+In%3A+Calzolari+N%2C+Choukri+K%2C+Maegaard+B%2C+Mariani+J%2C+Odijk+J%2C+Piperid)
9. Available from: https://aclanthology.org/L08-1226/. Penedo G, Kydlı́ček H, allal LB, Lozhkov A, Mitchell M, Raffel C, et al.: [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/). Available from: https://arxiv.org/abs/2406.17557. Delobelle P, Winters T, Berendt B. RobBERT: a Dutch RoBERTa-based Language Model. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings; 2020. p. 3255–3265. [↗](https://arxiv.org/abs/2406.17557)
10. Menger V, Scheepers F, van Wijk LM, Spruit M. DEDUCE: A pattern matching method for automatic de-identification of Dutch medical text. Telematics and Informatics. 2018;35(4):727 – 736. https://doi.org/https://doi.org/10.1016/j.tele.2017.08.002. [↗](https://doi.org/10.1016/j.tele.2017.08.002)
11. Dernoncourt F, Lee JY, Uzuner O, Szolovits P. De-identification of patient notes with recurrent neural networks. Journal of the American Medical Informatics Association. 2017;24(3):596–606. [find](https://scholar.google.com/scholar?q=Dernoncourt+F%2C+Lee+JY%2C+Uzuner+O%2C+Szolovits+P%2E+De%2Didentification+of+patient+notes+with+recurrent+neural+networks%2E+Journal+of+the+American+Medical+Informatics+Association%2E+2017%3B24%283%29%3A596%E2%80%93606%2E)
12. Team N, Costa-jussà MR, Cross J, Çelebi O, Elbayad M, Heafield K, et al.: No Language Left Behind: Scaling Human-Centered Machine Translation. Available from: https://arxiv.org/abs/2207.04672. Tiedemann J, Thottingal S. the World. OPUS-MT – Building open translation services for In: Proceedings of the 22nd Annual Conference of the European Association for Machine Translation. Lisboa, Portugal: European Association for Machine Translation; 2020. p. 479–480. Available from: https://aclanthology.org/2020.eamt-1.61. [↗](https://arxiv.org/abs/2207.04672)
13. Wang X, Chen N, Chen J, Wang Y, Zhen G, Zhang C, et al.: Apollo: A Lightweight Multilingual Medical LLM towards Democratizing Medical AI to 6B People. Available from: https://arxiv.org/abs/2403.03640. Johnson AE, Pollard TJ, Shen L, Lehman LwH, Feng M, Ghassemi M, et al. [MIMIC-III, a freely accessible critical care database](https://md.research.elamur.ai/free/en/papers/johnson-2016-mimic-iii-freely-accessible-critical/). Scientific data. 2016;3(1):1–9. [↗](https://arxiv.org/abs/2403.03640)
14. Johnson AE, Bulgarelli L, Shen L, Gayles A, Shammout A, Horng S, et al. [MIMIC-IV, a freely accessible electronic health record dataset](https://md.research.elamur.ai/free/en/papers/johnson-2023-mimic-iv-freely-accessible-electronic/). Scientific data. 2023;10(1):1.
15. Remy F, Demuynck K, Demeester T. Automatic Glossary of Clinical Terminology: a Large-Scale Dictionary of Biomedical Definitions Generated from Ontological Knowledge. In: Demner-fushman D, Ananiadou S, Cohen K, editors. The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks. Toronto, Canada: Association for Computational Linguistics; 2023. p. 265–272. [find](https://scholar.google.com/scholar?q=Remy+F%2C+Demuynck+K%2C+Demeester+T%2E+Automatic+Glossary+of+Clinical+Terminology%3A+a+Large%2DScale+Dictionary+of+Biomedical+Definitions+Generated+from+Ontological+Knowledge%2E+In%3A+Demner%2Dfushman+D%2C+Ananiadou+S%2C)
16. Available from: https://aclanthology.org/2023.bionlp-1.23. Remy F, Demuynck K, Demeester T. BioLORD: Learning Ontological Representations from Definitions for Biomedical Concepts and their Textual Descriptions. In: Findings of the Association for Computational Linguistics: EMNLP 2022; 2022. p. 1454–1465. [↗](https://aclanthology.org/2023.bionlp-1.23/)
17. Remy F, Demuynck K, Demeester T. BioLORD-2023: Semantic textual representations fusing llm and clinical knowledge graph insights. arXiv:231116075. 2023;. arXiv preprint Chen Z, Cano AH, Romanou A, Bonnet A, Matoba K, Salvi F, et al.: [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2023-meditron-70b-scaling-medical-pretraining/). Available from: https://arxiv.org/abs/2311.16079. [↗](https://arxiv.org/abs/2311.16079)
18. Ben Abacha A, Shivade C, Demner-Fushman D. Overview of the MEDIQA 2019 [find](https://scholar.google.com/scholar?q=Ben+Abacha+A%2C+Shivade+C%2C+Demner%2DFushman+D%2E+Overview+of+the+MEDIQA+2019)
19. Shared Task on Textual Inference, Question Entailment and Question Answering. In: ACL-BioNLP 2019; 2019. . [find](https://scholar.google.com/scholar?q=Shared+Task+on+Textual+Inference%2C+Question+Entailment+and+Question+Answering%2E+In%3A+ACL%2DBioNLP+2019%3B+2019%2E+%2E)
20. Jin D, Pan E, Oufattole N, Weng WH, Fang H, Szolovits P. [What disease does this patient have? a large-scale open domain question answering dataset from medical exams](https://md.research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/). Applied Sciences. 2021;11(14):6421.
21. Hendrycks D, Burns C, Basart S, Zou A, Mazeika M, Song D, et al. [Measuring Massive Multitask Language Understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In: International Conference on Learning Representations; 2020. .
22. Han T, Adams LC, Papaioannou JM, Grundmann P, Oberhauser T, Figueroa A, et al. MedAlpaca–an open-source collection of medical conversational AI models and training data. arXiv preprint arXiv:230408247. 2023;. — [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://md.research.elamur.ai/free/en/papers/han-2023-medalpaca-open-source-collection-medical/)
23. Krithara A, Nentidis A, Bougiatiotis K, Paliouras G. BioASQ-QA: A manually curated corpus for Biomedical Question Answering. Scientific Data. 2023;10(1):170. [find](https://scholar.google.com/scholar?q=Krithara+A%2C+Nentidis+A%2C+Bougiatiotis+K%2C+Paliouras+G%2E+BioASQ%2DQA%3A+A+manually+curated+corpus+for+Biomedical+Question+Answering%2E+Scientific+Data%2E+2023%3B10%281%29%3A170%2E)
24. Zheng Q, Abdullah S, Rawal S, Zakka C, Ostmeier S, Purk M, et al. MIRIAD: Augmenting LLMs with millions of medical query-response pairs. arXiv preprint arXiv:250606091. 2025;. [find](https://scholar.google.com/scholar?q=Zheng+Q%2C+Abdullah+S%2C+Rawal+S%2C+Zakka+C%2C+Ostmeier+S%2C+Purk+M%2C+et+al%2E+MIRIAD%3A+Augmenting+LLMs+with+millions+of+medical+query%2Dresponse+pairs%2E+arXiv+preprint+arXiv%3A250606091%2E+2025%3B%2E)

*Updated 29 October 2026*