---
title: "PortOldBERT: Portuguese Historical Language Models (2026)"
description: "Historical texts differ from modern ones in vocabulary, meaning, morphology and especially spelling, which was not… 64 references."
authors: ["Tomas Freitas Osorio", "Henrique Lopes Cardoso"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.eacl-long.123/"
doi: "10.18653/v1/2026.eacl-long.123"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/osorio-2026-portoldbert-portuguese-historical-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/osorio-2026-portoldbert-portuguese-historical-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/39/) / PortOldBERT: Portuguese Historical Language Models

# PortOldBERT: Portuguese Historical Language Models

**Authors:** Tomas Freitas Osorio and [Henrique Lopes Cardoso](https://md.research.elamur.ai/free/en/authors/henrique-lopes-cardoso/) · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-long.123/) · [PDF](https://aclanthology.org/2026.eacl-long.123.pdf) · [DOI 10.18653/v1/2026.eacl-long.123](https://doi.org/10.18653/v1/2026.eacl-long.123) · [OpenAlex](https://openalex.org/W7140089395)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary

Historical texts differ from modern ones in vocabulary, meaning, morphology and especially spelling, which was not standardised before the 18th–19th centuries. Pretrained encoders for such texts exist mainly for English and a few other languages. None existed for historical Portuguese. The authors address this with PortOldBERT. Instead of training from scratch, they continued pre-training Albertina, a DeBERTa-based encoder for contemporary Portuguese, on eighteen freely available corpora covering the 12th to the 20th centuries. They produced one model per historical period and one combined model, PortOldBERT-All. These were compared with the original Albertina pt-PT and pt-BR models on:

- pseudo-perplexity
- NER and PoS tagging
- two new benchmarks built from OCR–transcription pairs: WER prediction and OCR error detection

The combined model fits historical text best and leads clearly on the OCR-based tasks. Its gains on NER and PoS are smaller, likely because evaluation data is scarce and largely auto-annotated. The models and training corpora are publicly released, giving historical Portuguese NLP its first dedicated encoder.

## Key points

- **Training data and tokenizer:** The pre-training data contains about 48.5M training tokens and 3.9M test tokens, grouped into six periods (\<16th, 16th–18th, two halves each of the 19th and 20th centuries). The tokenizer produced zero OOV tokens, with a subword fertility rate between 1.96 and 2.15.
- **Downstream benchmarks:** NER was evaluated on BDCamões, ELTeC and PPM, and PoS tagging on ELTeC, Colonia, Tycho and BDCamões. Only the PPM test document has manual annotation, and that is where PortOldBERT-All's advantage is clearest.
- **WER prediction:** This task used PortOldBERT-All-EP, a variant trained without overlapping PORTO documents. It reached R² 0.796 versus 0.639 for Albertina pt-BR and 0.597 for pt-PT, and correlation 0.900 versus 0.815.
- **OCR error detection:** PortOldBERT-All-EP scored F1 0.888, against 0.865 for Albertina pt-PT and 0.861 for pt-BR.
- **Availability:** The models are on HuggingFace (LIACC/PortOldBERT) under CC BY-NC 4.0, together with the training corpora (LIACC/HistoricalPortugueseCorpora).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Tomas Freitas Osorio and Henrique Lopes Cardoso. 2026. PortOldBERT: Portuguese Historical Language Models. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{osorio2026portoldbert,
  title = {{PortOldBERT: Portuguese Historical Language Models}},
  author = {Tomas Freitas Osorio and Henrique Lopes Cardoso},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.123},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.123}
}
```

- [← previous Portho: A Corpus-Based Resource of Orthographic Neighbors in European Portuguese](https://md.research.elamur.ai/free/en/papers/ribeiro-2026-portho-corpus-based-resource-orthographic/)
- [next → POS Tagging in Low-Resource Maithili Language: Specific Challenges and Nuances](https://md.research.elamur.ai/free/en/papers/priya-2026-pos-tagging-low-resource-maithili/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/39/)

## References

64 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Reem Alatrash, Dominik Schlechtweg, Jonas Kuhn, and Sabine Schulte im Walde. 2020. [CCOHA: Clean Corpus of Historical American English](https://md.research.elamur.ai/free/en/papers/alatrash-2020-ccoha-clean-corpus-historical-american/). In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6958–6966, Marseille, France. European Language Resources Association.
2. Chancelaria de D. Afonso III: documentos em português. 2023. [https://hdl.handle.net/21.11129/0000-000D-FE7C-B](https://hdl.handle.net/21.11129/0000-000D-FE7C-B). Accessed: 16-5-2023. [↗](https://hdl.handle.net/21.11129/0000-000D-FE7C-B)
3. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pre-training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V%2E+Le%2C+and+Christopher+D%2E+Manning%2E+2020%2E+ELECTRA%3A+pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+In+8th+International+Conference+on+Learning+R)
4. Comunidade dos Países de Língua Portuguesa (CPLP). 2023. [https://www.cplp.org/id-2597.aspx](https://www.cplp.org/id-2597.aspx). Accessed: 2022-10-26. [↗](https://www.cplp.org/id-2597.aspx)
5. Corpus Eletrônico de Documentos Históricos do Sertão. 2023. [http://www5.uefs.br/cedohs/view/home.html](http://www5.uefs.br/cedohs/view/home.html). Accessed: 16-5-2023. [↗](http://www5.uefs.br/cedohs/view/home.html)
6. Arquivo dos Açores. 2023. [https://hdl.handle.net/21.11129/0000-000D-F8C0-2](https://hdl.handle.net/21.11129/0000-000D-F8C0-2). Accessed: 16-5-2023. [↗](https://hdl.handle.net/21.11129/0000-000D-F8C0-2)
7. Corpus Histórico da Linguagem da Medicina em Português (Século XVIII): Terminologia Diacrônica e Humanidades Digitais. 2023. [https://sites.google.com/view/projeto38597](https://sites.google.com/view/projeto38597). Accessed: 16-5-2023. [↗](https://sites.google.com/view/projeto38597)
8. Arquivo Pessoa. 2023. [http://arquivopessoa.net/](http://arquivopessoa.net/). Accessed: 15-05-2023. [↗](http://arquivopessoa.net/)
9. CTACorpus. 2023. [http://teitok.clul.ul.pt/cta/](http://teitok.clul.ul.pt/cta/). Accessed: 13-12-2022. [↗](http://teitok.clul.ul.pt/cta/)
10. As Memórias Paroquiais de 1758. 2023. [http://www.cidehusdigital.uevora.pt/portugal1758](http://www.cidehusdigital.uevora.pt/portugal1758). Accessed: 15-05-2023. [↗](http://www.cidehusdigital.uevora.pt/portugal1758)
11. David M. Eberhard, Gary F. Simons, and Charles D. Fennig. 2023. Ethnologue: Languages of the World, 26 edition. SIL International, Dallas. [find](https://scholar.google.com/scholar?q=David+M%2E+Eberhard%2C+Gary+F%2E+Simons%2C+and+Charles+D%2E+Fennig%2E+2023%2E+Ethnologue%3A+Languages+of+the+World%2C+26+edition%2E+SIL+International%2C+Dallas%2E)
12. David Bamman and Patrick J. Burns. 2020. Latin bert: A contextual language model for classical philology. ArXiv, abs/2009.10053. [find](https://scholar.google.com/scholar?q=David+Bamman+and+Patrick+J%2E+Burns%2E+2020%2E+Latin+bert%3A+A+contextual+language+model+for+classical+philology%2E+ArXiv%2C+abs%2F2009%2E10053%2E)
13. Margarida Falcão, Mariana Dias, and Carla Teixeira Lopes. 2022. Manual transcriptions of typewritten digital representations of portuguese cultural heritage documents from the 20th century. [https://rdm.inesctec.pt/dataset/cs-2022-005](https://rdm.inesctec.pt/dataset/cs-2022-005). [↗](https://rdm.inesctec.pt/dataset/cs-2022-005)
14. Christin Beck and Marisa Köllner. 2023. GHisBERT – training BERT from scratch for lexical semantic investigations across historical German language stages. In Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change, pages 33–45, Singapore. Association for Computatio… [find](https://scholar.google.com/scholar?q=Christin+Beck+and+Marisa+K%C3%B6llner%2E+2023%2E+GHisBERT+%E2%80%93+training+BERT+from+scratch+for+lexical+semantic+investigations+across+historical+German+language+stages%2E+In+Proceedings+of+the+4th+Workshop+on+Comput)
15. Eckhard Bick. 2006. Functional aspects in portuguese ner. In Computational Processing of the Portuguese Language, pages 80–89, Berlin, Heidelberg. Springer Berlin Heidelberg. [find](https://scholar.google.com/scholar?q=Eckhard+Bick%2E+2006%2E+Functional+aspects+in+portuguese+ner%2E+In+Computational+Processing+of+the+Portuguese+Language%2C+pages+80%E2%80%9389%2C+Berlin%2C+Heidelberg%2E+Springer+Berlin+Heidelberg%2E)
16. Eckhard Bick. 2014. PALAVRAS - A Constraint Grammar-Based Parsing System for Portuguese, pages 279–302. Bloomsbury Academic. [find](https://scholar.google.com/scholar?q=Eckhard+Bick%2E+2014%2E+PALAVRAS+%2D+A+Constraint+Grammar%2DBased+Parsing+System+for+Portuguese%2C+pages+279%E2%80%93302%2E+Bloomsbury+Academic%2E)
17. Andreas Blank. 1999. Why do new meanings occur? A cognitive typology of the motivations for lexical semantic change, pages 61–90. De Gruyter Mouton, Berlin, Boston. [find](https://scholar.google.com/scholar?q=Andreas+Blank%2E+1999%2E+Why+do+new+meanings+occur%3F+A+cognitive+typology+of+the+motivations+for+lexical+semantic+change%2C+pages+61%E2%80%9390%2E+De+Gruyter+Mouton%2C+Berlin%2C+Boston%2E)
18. António Branco and João Ricardo Silva. 2006. A suite of shallow processing tools for Portuguese: LX-suite. In Demonstrations, pages 179–182. [find](https://scholar.google.com/scholar?q=Ant%C3%B3nio+Branco+and+Jo%C3%A3o+Ricardo+Silva%2E+2006%2E+A+suite+of+shallow+processing+tools+for+Portuguese%3A+LX%2Dsuite%2E+In+Demonstrations%2C+pages+179%E2%80%93182%2E)
19. Jean-Baptiste Camps, Simon Gabay, Paul Fièvre, Thibault Clérice, and Florian Cafiero. 2021. Corpus and Models for Lemmatisation and POS-tagging of Classical French Theatre. Journal of Data Mining & Digital Humanities, 2021. [find](https://scholar.google.com/scholar?q=Jean%2DBaptiste+Camps%2C+Simon+Gabay%2C+Paul+Fi%C3%A8vre%2C+Thibault+Cl%C3%A9rice%2C+and+Florian+Cafiero%2E+2021%2E+Corpus+and+Models+for+Lemmatisation+and+POS%2Dtagging+of+Classical+French+Theatre%2E+Journal+of+Data+Mining+%26+Di)
20. Lizhou Fan, Lingyao Li, Zihui Ma, Sanggyu Lee, Huizi Yu, and Libby Hemphill. 2024. A bibliometric review of large language models research from 2017 to 2023. ACM Trans. Intell. Syst. Technol. Just Accepted. [find](https://scholar.google.com/scholar?q=Lizhou+Fan%2C+Lingyao+Li%2C+Zihui+Ma%2C+Sanggyu+Lee%2C+Huizi+Yu%2C+and+Libby+Hemphill%2E+2024%2E+A+bibliometric+review+of+large+language+models+research+from+2017+to+2023%2E+ACM+Trans%2E+Intell%2E+Syst%2E+Technol%2E+Just+Acc)
21. Joshua A. Fishman. 1964. Language maintenance and language shift as a field of inquiry. a definition of the field and suggestions for its further development. Linguistics, 2(9):32–70. [find](https://scholar.google.com/scholar?q=Joshua+A%2E+Fishman%2E+1964%2E+Language+maintenance+and+language+shift+as+a+field+of+inquiry%2E+a+definition+of+the+field+and+suggestions+for+its+further+development%2E+Linguistics%2C+2%289%29%3A32%E2%80%9370%2E)
22. Tomás Freitas Osório and Henrique Lopes Cardoso. 2025. Portuguese post-ocr resources for text optimisation. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, CIKM ’25, page 6361–6366, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Tom%C3%A1s+Freitas+Os%C3%B3rio+and+Henrique+Lopes+Cardoso%2E+2025%2E+Portuguese+post%2Docr+resources+for+text+optimisation%2E+In+Proceedings+of+the+34th+ACM+International+Conference+on+Information+and+Knowledge+Managem)
23. Simon Gabay, Pedro Ortiz Suarez, Alexandre Bartz, Alix Chagué, Rachel Bawden, Philippe Gambette, and Benoît Sagot. 2022. From FreEM to d’AlemBERT: a large corpus and a language model for early Modern French. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3367–33… [find](https://scholar.google.com/scholar?q=Simon+Gabay%2C+Pedro+Ortiz+Suarez%2C+Alexandre+Bartz%2C+Alix+Chagu%C3%A9%2C+Rachel+Bawden%2C+Philippe+Gambette%2C+and+Beno%C3%AEt+Sagot%2E+2022%2E+From+FreEM+to+d%E2%80%99AlemBERT%3A+a+large+corpus+and+a+language+model+for+early+Modern+)
24. Charlotte Galves. 2018. The tycho brahe corpus of historical portuguese: Methodology and results. Linguistic Variation, 18:49–73. [find](https://scholar.google.com/scholar?q=Charlotte+Galves%2E+2018%2E+The+tycho+brahe+corpus+of+historical+portuguese%3A+Methodology+and+results%2E+Linguistic+Variation%2C+18%3A49%E2%80%9373%2E)
25. GMHP. 2023. [http://www.usp.br/gmhp/CorpI.html](http://www.usp.br/gmhp/CorpI.html). Accessed: 14-06-2022. [↗](http://www.usp.br/gmhp/CorpI.html)
26. Mariana Gomes, Ana Guilherme, Leonor Tavares, and Rita Marquilhas. 2012. Project FLY: a multidisciplinary project within linguistics. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2833–2837, Istanbul, Turkey. European Language Resources A… [find](https://scholar.google.com/scholar?q=Mariana+Gomes%2C+Ana+Guilherme%2C+Leonor+Tavares%2C+and+Rita+Marquilhas%2E+2012%2E+Project+FLY%3A+a+multidisciplinary+project+within+linguistics%2E+In+Proceedings+of+the+Eighth+International+Conference+on+Language+)
27. Sara Grilo, Márcia Bolrinha, João Silva, Rui Vaz, and António Branco. 2020. The bdcamões collection of portuguese literary documents: a research resource for language technology and digital humanities. Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 849– 85… [find](https://scholar.google.com/scholar?q=Sara+Grilo%2C+M%C3%A1rcia+Bolrinha%2C+Jo%C3%A3o+Silva%2C+Rui+Vaz%2C+and+Ant%C3%B3nio+Branco%2E+2020%2E+The+bdcam%C3%B5es+collection+of+portuguese+literary+documents%3A+a+research+resource+for+language+technology+and+digital+humanities)
28. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. A… [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Ana+Marasovic%CC%81%2C+Swabha+Swayamdipta%2C+Kyle+Lo%2C+Iz+Beltagy%2C+Doug+Downey%2C+and+Noah+A%2E+Smith%2E+2020%2E+Don%E2%80%99t+stop+pretraining%3A+Adapt+language+models+to+domains+and+tasks%2E+In+Proceedings+of+)
29. Michel Généreux, Iris Hendrickx, and Amália Mendes. 2012. A large portuguese corpus on-line: Cleaning and preprocessing. Computational Processing of the Portuguese Language. PROPOR, pages 113–120. [find](https://scholar.google.com/scholar?q=Michel+G%C3%A9n%C3%A9reux%2C+Iris+Hendrickx%2C+and+Am%C3%A1lia+Mendes%2E+2012%2E+A+large+portuguese+corpus+on%2Dline%3A+Cleaning+and+preprocessing%2E+Computational+Processing+of+the+Portuguese+Language%2E+PROPOR%2C+pages+113%E2%80%93120%2E)
30. William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. [Diachronic word embeddings reveal statistical laws of semantic change](https://md.research.elamur.ai/free/en/papers/hamilton-2016-diachronic-word-embeddings-reveal-statistical/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489–1501, Berlin, Germany. Association for Com…
31. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022. [Towards a unified view of parameter-efficient transfer learning](https://md.research.elamur.ai/free/en/papers/he-2021-towards-unified-view-parameter-efficient/). In International Conference on Learning Representations.
32. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. [Deberta: Decoding-enhanced bert with disentangled attention](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/). Preprint, arXiv:2006.03654.
33. Kasra Hosseini, Kaspar Beelen, Giovanni Colavizza, and Mariona Coll Ardanuy. 2021. [Neural language models for nineteenth-century english](https://md.research.elamur.ai/free/en/papers/hosseini-2021-neural-language-models-nineteenth-century/). Journal of Open Humanities Data.
34. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Ma… [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+De+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+NLP%2E+In+)
35. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). Preprint, arXiv:2106.09685.
36. Paul Kerswill. 2006. Migration and Language, volume Volume 3. De Gruyter Mouton, Berlin • New York. [find](https://scholar.google.com/scholar?q=Paul+Kerswill%2E+2006%2E+Migration+and+Language%2C+volume+Volume+3%2E+De+Gruyter+Mouton%2C+Berlin+%E2%80%A2+New+York%2E)
37. Leonard Konle and Fotis Jannidis. 2020. Domain and task adaptive pretraining for language models. In Workshop on Computational Humanities Research. [find](https://scholar.google.com/scholar?q=Leonard+Konle+and+Fotis+Jannidis%2E+2020%2E+Domain+and+task+adaptive+pretraining+for+language+models%2E+In+Workshop+on+Computational+Humanities+Research%2E)
38. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). Preprint, arXiv:1907.11692.
39. Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021. Compacter: Efficient low-rank hypercomplex adapter layers. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Rabeeh+Karimi+Mahabadi%2C+James+Henderson%2C+and+Sebastian+Ruder%2E+2021%2E+Compacter%3A+Efficient+low%2Drank+hypercomplex+adapter+layers%2E+In+Advances+in+Neural+Information+Processing+Systems%2E)
40. Enrique Manjavacas and Lauren Fonteyn. 2022. Adapting vs. Pre-training Language Models for Historical Languages. Journal of Data Mining & Digital Humanities, NLP4DH. [find](https://scholar.google.com/scholar?q=Enrique+Manjavacas+and+Lauren+Fonteyn%2E+2022%2E+Adapting+vs%2E+Pre%2Dtraining+Language+Models+for+Historical+Languages%2E+Journal+of+Data+Mining+%26+Digital+Humanities%2C+NLP4DH%2E)
41. Enrique Manjavacas Arevalo and Lauren Fonteyn. 2021. [MacBERTh: Development and evaluation of a historically pre-trained language model for English (1450-1950)](https://md.research.elamur.ai/free/en/papers/arevalo-2021-macberth-development-evaluation-historically-pre/). In Proceedings of the Workshop on Natural Language Processing for Digital Humanities, pages 23–36, NIT Silchar, India. NLP Association of In…
42. Enrique Manjavacas Arevalo and Lauren Fonteyn. 2022. Non-parametric word sense disambiguation for historical languages. In Proceedings of the 2nd International Workshop on Natural Language Processing for Digital Humanities, pages 123–134, Taipei, Taiwan. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Enrique+Manjavacas+Arevalo+and+Lauren+Fonteyn%2E+2022%2E+Non%2Dparametric+word+sense+disambiguation+for+historical+languages%2E+In+Proceedings+of+the+2nd+International+Workshop+on+Natural+Language+Processing+)
43. Tomás Freitas Osório and Henrique Lopes Cardoso. 2024. Historical Portuguese corpora: a survey. Language Resources and Evaluation. [find](https://scholar.google.com/scholar?q=Tom%C3%A1s+Freitas+Os%C3%B3rio+and+Henrique+Lopes+Cardoso%2E+2024%2E+Historical+Portuguese+corpora%3A+a+survey%2E+Language+Resources+and+Evaluation%2E)
44. Alessio Palmero Aprosio, Stefano Menini, and Sara Tonelli. 2022. BERToldo, the historical BERT for Italian. In Proceedings of the Second Workshop on Language Technologies for Historical and Ancient Languages, pages 68–72, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Alessio+Palmero+Aprosio%2C+Stefano+Menini%2C+and+Sara+Tonelli%2E+2022%2E+BERToldo%2C+the+historical+BERT+for+Italian%2E+In+Proceedings+of+the+Second+Workshop+on+Language+Technologies+for+Historical+and+Ancient+La)
45. Jose Ramom Pichel Campos, Pablo Gamallo, and Iñaki Alegria. 2018. Measuring language distance among historical varieties using perplexity. application to European Portuguese. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), pages 145–155, Sant… [find](https://scholar.google.com/scholar?q=Jose+Ramom+Pichel+Campos%2C+Pablo+Gamallo%2C+and+I%C3%B1aki+Alegria%2E+2018%2E+Measuring+language+distance+among+historical+varieties+using+perplexity%2E+application+to+European+Portuguese%2E+In+Proceedings+of+the+Fif)
46. Project Gutenberg. 2023. [https://www.gutenberg.org/browse/languages/pt](https://www.gutenberg.org/browse/languages/pt). Accessed: 15-05-2023. [↗](https://www.gutenberg.org/browse/languages/pt)
47. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). J. Mach. Learn. Res., 21(1).
48. Gael Vaamonde, Ana Luísa Costa, Rita Marquilhas, Clara Pinto, and Fernanda Pratas. 2014. Post scriptum: Archivo digital de escritura cotidiana. Humanidades Digitales: desafíos, logros y perspectivas de futuro, pages 473–482. [find](https://scholar.google.com/scholar?q=Gael+Vaamonde%2C+Ana+Lu%C3%ADsa+Costa%2C+Rita+Marquilhas%2C+Clara+Pinto%2C+and+Fernanda+Pratas%2E+2014%2E+Post+scriptum%3A+Archivo+digital+de+escritura+cotidiana%2E+Humanidades+Digitales%3A+desaf%C3%ADos%2C+logros+y+perspectivas+d)
49. Frederick Riemenschneider and Anette Frank. 2023. Exploring large language models for classical philology. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15181–15199, Toronto, Canada. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Frederick+Riemenschneider+and+Anette+Frank%2E+2023%2E+Exploring+large+language+models+for+classical+philology%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28V)
50. Renata Vieira, Fernanda Olival, Helena Freire Cameron, Joaquim Santos, Ofélia Sequeira, and Ivo Santos. 2021. Enriching the 1758 portuguese parish memories (alentejo) with named entities. Journal of Open Humanities Data. [find](https://scholar.google.com/scholar?q=Renata+Vieira%2C+Fernanda+Olival%2C+Helena+Freire+Cameron%2C+Joaquim+Santos%2C+Of%C3%A9lia+Sequeira%2C+and+Ivo+Santos%2E+2021%2E+Enriching+the+1758+portuguese+parish+memories+%28alentejo%29+with+named+entities%2E+Journal+of+O)
51. Shruti Rijhwani, Daisy Rosenblum, Antonios Anastasopoulos, and Graham Neubig. 2021. Lexically aware semi-supervised learning for OCR post-correction. Transactions of the Association for Computational Linguistics, 9:1285–1302. [find](https://scholar.google.com/scholar?q=Shruti+Rijhwani%2C+Daisy+Rosenblum%2C+Antonios+Anastasopoulos%2C+and+Graham+Neubig%2E+2021%2E+Lexically+aware+semi%2Dsupervised+learning+for+OCR+post%2Dcorrection%2E+Transactions+of+the+Association+for+Computational+)
52. João Rodrigues, Luís Gomes, João Silva, António Branco, Rodrigo Santos, Henrique Lopes Cardoso, and Tomás Osório. 2023. Advancing Neural Encoding of Portuguese with Transformer Albertina PT-*, page 441–453. Springer Nature Switzerland. [find](https://scholar.google.com/scholar?q=Jo%C3%A3o+Rodrigues%2C+Lu%C3%ADs+Gomes%2C+Jo%C3%A3o+Silva%2C+Ant%C3%B3nio+Branco%2C+Rodrigo+Santos%2C+Henrique+Lopes+Cardoso%2C+and+Tom%C3%A1s+Os%C3%B3rio%2E+2023%2E+Advancing+Neural+Encoding+of+Portuguese+with+Transformer+Albertina+PT%2D%2A%2C+page+44)
53. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. [How good is your tokenizer? on the monolingual performance of multilingual language models](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join…
54. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. [Masked language model scoring](https://md.research.elamur.ai/free/en/papers/salazar-2020-masked-language-model-scoring/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712, Online. Association for Computational Linguistics.
55. Zhonghao Wang, Zijia Lu, Bo Jin, and Haiying Deng. 2023. Mediagpt : A large language model for chinese media. Preprint, arXiv:2307.10930. — [arXiv:2307.10930](https://arxiv.org/abs/2307.10930) [find](https://scholar.google.com/scholar?q=Zhonghao+Wang%2C+Zijia+Lu%2C+Bo+Jin%2C+and+Haiying+Deng%2E+2023%2E+Mediagpt+%3A+A+large+language+model+for+chinese+media%2E+Preprint%2C+arXiv%3A2307%2E10930%2E+%E2%80%94+%5BarXiv%3A2307%2E10930%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2307%2E10930%29)
56. Maria Francisca Xavier. 2016. O CIPM – Corpus In-formatizado do Português Medieval, fonte de um Dicionário exaustivo, pages 137–156. De Gruyter, Berlin, Boston. [find](https://scholar.google.com/scholar?q=Maria+Francisca+Xavier%2E+2016%2E+O+CIPM+%E2%80%93+Corpus+In%2Dformatizado+do+Portugu%C3%AAs+Medieval%2C+fonte+de+um+Dicion%C3%A1rio+exaustivo%2C+pages+137%E2%80%93156%2E+De+Gruyter%2C+Berlin%2C+Boston%2E)
57. Marcos Zampieri and Martin Becker. 2013. Colonia: Corpus of historical portuguese. Non-Standard Data Sources in Corpus-based Research. ZSM-Studien Series - Vol. 5. [find](https://scholar.google.com/scholar?q=Marcos+Zampieri+and+Martin+Becker%2E+2013%2E+Colonia%3A+Corpus+of+historical+portuguese%2E+Non%2DStandard+Data+Sources+in+Corpus%2Dbased+Research%2E+ZSM%2DStudien+Series+%2D+Vol%2E+5%2E)
58. Shitou Zhang, Jingrui Hou, Siyuan Peng, Zuchao Li, Qibiao Hu, and Ping Wang. 2023. Arcgpt: A large language model tailored for real-world archival applications. Preprint, arXiv:2307.14852. — [arXiv:2307.14852](https://arxiv.org/abs/2307.14852) [find](https://scholar.google.com/scholar?q=Shitou+Zhang%2C+Jingrui+Hou%2C+Siyuan+Peng%2C+Zuchao+Li%2C+Qibiao+Hu%2C+and+Ping+Wang%2E+2023%2E+Arcgpt%3A+A+large+language+model+tailored+for+real%2Dworld+archival+applications%2E+Preprint%2C+arXiv%3A2307%2E14852%2E+%E2%80%94+%5BarXiv%3A23)
59. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. [A survey of l…](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)
60. A Diana Santos. 2021. Portuguese novel collection (eltec-por). European Literary Text Collection (ELTeC). [find](https://scholar.google.com/scholar?q=A+Diana+Santos%2E+2021%2E+Portuguese+novel+collection+%28eltec%2Dpor%29%2E+European+Literary+Text+Collection+%28ELTeC%29%2E)
61. B B Schieffelin and E Ochs. 1986. Language socialization. Annual Review of Anthropology, 15(1):163– 191. [find](https://scholar.google.com/scholar?q=B+B+Schieffelin+and+E+Ochs%2E+1986%2E+Language+socialization%2E+Annual+Review+of+Anthropology%2C+15%281%29%3A163%E2%80%93+191%2E)
62. Stefan Schweter, Luisa März, Katharina Schmid, and Erion Çano. 2022. hmbert: Historical multilingual language models for named entity recognition. In CLEF 2022: Conference and Labs of the Evaluation Forum, pages 1109–1129, Bologna, Italy. [find](https://scholar.google.com/scholar?q=Stefan+Schweter%2C+Luisa+M%C3%A4rz%2C+Katharina+Schmid%2C+and+Erion+%C3%87ano%2E+2022%2E+hmbert%3A+Historical+multilingual+language+models+for+named+entity+recognition%2E+In+CLEF+2022%3A+Conference+and+Labs+of+the+Evaluation+F)
63. Murray Shanahan. 2024. Talking about large language models. Commun. ACM, 67(2):68–79. [find](https://scholar.google.com/scholar?q=Murray+Shanahan%2E+2024%2E+Talking+about+large+language+models%2E+Commun%2E+ACM%2C+67%282%29%3A68%E2%80%9379%2E)
64. Pranaydeep Singh, Gorik Rutten, and Els Lefever. 2021. A pilot study for BERT language modelling and morphological analysis for ancient and medieval Greek. In Proceedings of the 5th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature,… [find](https://scholar.google.com/scholar?q=Pranaydeep+Singh%2C+Gorik+Rutten%2C+and+Els+Lefever%2E+2021%2E+A+pilot+study+for+BERT+language+modelling+and+morphological+analysis+for+ancient+and+medieval+Greek%2E+In+Proceedings+of+the+5th+Joint+SIGHUM+Works)

*Updated 1 November 2026*