---
title: "Neural Language Models for Nineteenth-Century English — paper by Hosseini et al. (2021)"
description: "NLP tools built on present-day English can misrepresent older documents, because vocabulary and usage shift over… 12 references, cited by 17 papers here."
authors: ["Kasra Hosseini", "Kaspar Beelen", "Giovanni Colavizza", "Mariona Coll Ardanuy"]
year: 2021
venue: "arXiv:2105.11321"
source: "https://arxiv.org/abs/2105.11321"
doi: "10.48550/arxiv.2105.11321"
arxiv: "2105.11321"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hosseini-2021-neural-language-models-nineteenth-century/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hosseini-2021-neural-language-models-nineteenth-century/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/6/) / Neural Language Models for Nineteenth-Century English

# Neural Language Models for Nineteenth-Century English

**Authors:** Kasra Hosseini, Kaspar Beelen, [Giovanni Colavizza](https://md.research.elamur.ai/free/en/authors/giovanni-colavizza/) and Mariona Coll Ardanuy · **Year:** 2021 · **Venue:** [arXiv:2105.11321](https://md.research.elamur.ai/free/en/venues/journal-of-open-humanities-data/) ([Journal of Open Humanities Data 2021](https://md.research.elamur.ai/free/en/venues/journal-of-open-humanities-data/2021/)) · **Publisher:** [Ubiquity Press](https://md.research.elamur.ai/free/en/publishers/ubiquity-press/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2105.11321) · [PDF](https://arxiv.org/pdf/2105.11321) · [DOI 10.48550/arxiv.2105.11321](https://doi.org/10.48550/arxiv.2105.11321) · [arXiv 2105.11321](https://arxiv.org/abs/2105.11321) · [OpenAlex](https://openalex.org/W3164937913)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [17](https://md.research.elamur.ai/free/en/papers/hosseini-2021-neural-language-models-nineteenth-century/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
NLP tools built on present-day English can misrepresent older documents, because vocabulary and usage shift over time. This paper comes from the Living with Machines project at The Alan Turing Institute. It releases a family of language models adapted to nineteenth-century English. The training material is the Microsoft British Library Corpus (MBL), a set of public-domain digitized books. The authors applied light normalization, kept only English books using spaCy, and segmented the text with syntok. The result was 47,685 books and roughly 5.1 billion tokens dated 1760–1900. Four architectures were trained: static embeddings (word2vec, fastText), a character-level Flair LSTM, and BERT. Besides full-corpus versions, there are pre-1850 static models and four BERT checkpoints. Each BERT checkpoint was produced by continuing fine-tuning from the previous one on the next time slice. No intrinsic benchmark is given. Quality is argued from earlier downstream studies in which the models helped on data from the same period. The collection, histLM, is openly licensed and gives historians and NLP researchers ready-made tools for studying diachronic change.

## Key points
- Corpus: MBL, digitized by the British Library with Microsoft. About 48K books were reduced to 47,685 English books (≈5.1B tokens) after ASCII conversion, punctuation repair, dehyphenation, spaCy language filtering and syntok sentence splitting.
- word2vec and fastText were trained in Gensim on lowercased text: skip-gram, 300 dimensions, minimum frequency 20, one epoch. There are full-corpus and pre-1850 variants.
- Flair is a forward character-level LSTM with one layer of hidden size 2048. It was trained for one epoch on 250-character sequences with the default character dictionary.
- BERT starts from 'BERT base uncased' and is fine-tuned with the MLM objective only, in chained stages:
  - The slices are pre-1850, 1850–1875, 1875–1890 and 1890–1900.
  - Each slice has ≈1.3B tokens, except ≈1.1B for the last.
  - Training used lr 1e-4, 2,000 warm-up steps, batch 5 per GPU on 4× K80, and kept the original tokenizer.
- The models are released as histLM on Zenodo under CC BY 4.0, with helper scripts on GitHub. Earlier evaluation was extrinsic: OCR-noise robustness, atypical animacy detection and targeted sense disambiguation.

## Interesting ideas
- Chained period adaptation: each checkpoint inherits from the previous era's model instead of from the modern base, giving a smooth sequence of comparable models across time.
- Freezing the tokenizer across all variants means any difference between models reflects learned representations, not segmentation artifacts.
- Time slices were chosen mainly to balance token counts rather than to follow round calendar boundaries, so each slice model sees comparable amounts of data.
- The models helped mainly when the evaluation data came from the same period as the training text, which suggests checking how well training and test eras match.
- Minimal but targeted cleanup suits noisy historical scans: dehyphenating line-broken words, filtering by language and separating punctuation.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tree): gparse could be adapted step by step from modern treebanks toward older registers like fable translations, keeping each step as a checkpoint so the stage that changes a parse can be identified.
- **I2** (tokens): keep the en tokenizer fixed across domain or era variants so that tagger and lemma differences between them can be explained cleanly.
- **I3** (learn): split silver training data and evaluation buckets by comparable volume, so that comparing buckets reflects real domain effects rather than data size.
- **I4** (learn): evaluate on period-matched test sets and add an era-mismatched set as a negative control, to show whether gains really come from historical adaptation.
- **I5** (tokens): add a dehyphenation and language-filter pass before tokenizing OCR'd old tales, so that split words do not corrupt morphology and lemmas.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Amsterdam](https://md.research.elamur.ai/free/en/universities/university-of-amsterdam/) — University, Amsterdam, Netherlands · 89 works · 81,337 citations
- [Queen Mary University of London](https://md.research.elamur.ai/free/en/universities/queen-mary-university-of-london/) — University, London, United Kingdom · 39 works · 3,307 citations
- [The Alan Turing Institute](https://md.research.elamur.ai/free/en/labs/the-alan-turing-institute/) — Research institute, London, United Kingdom · 19 works · 3,608 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [📚 Literature and Literary Theory](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/literature-and-literary-theory/) · 340 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works

## Cite

Cite as: Kasra Hosseini, Kaspar Beelen, Giovanni Colavizza and Mariona Coll Ardanuy. 2021. Neural Language Models for Nineteenth-Century English. arXiv:2105.11321.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hosseini2021neural,
  title = {{Neural Language Models for Nineteenth-Century English}},
  author = {Kasra Hosseini, Kaspar Beelen, Giovanni Colavizza and Mariona Coll Ardanuy},
  year = {2021},
  doi = {10.48550/arxiv.2105.11321},
  eprint = {2105.11321},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2105.11321}
}
```

- [← previous Moroccan Dialect -Darija- Open Dataset](https://md.research.elamur.ai/free/en/papers/outchakoucht-2021-moroccan-dialect-darija-open-dataset/)
- [next → Robustifying Multi-hop QA through Pseudo-Evidentiality Training](https://md.research.elamur.ai/free/en/papers/lee-2021-robustifying-multi-hop-qa-through/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/6/)

## References

12 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Akbik, A., Bergmann, T., Blythe, D., Rasul, K., Schweter, S., & Vollgraf, R. (2019). Flair: An easy-to-use framework for state-of-the-art nlp. In NAACL 2019, 2019 annual conference of the north american chapter of the association for computational linguistics (demonstrations) (pp. 54–59). [our card](https://md.research.elamur.ai/free/en/papers/akbik-2019-flair-easy-use-framework-state/)
2. Azarbonyad, H., Dehghani, M., Beelen, K., Arkut, A., Marx, M., & Kamps, J. (2017). Words are malleable: Computing semantic shifts in political and media discourse. In Proceedings of the 2017 acm on conference on information and knowledge management (pp. 1509–1518). [↗](https://doi.org/10.48550/arxiv.1711.05603)
3. Beelen, K., Nanni, F., Coll Ardanuy, M., Hosseini, K., Tolfo, G., & McGillivray, B. (2021). When time makes sense: A historically-aware approach to targeted sense disambiguation. In Findings of acl-ijcnlp. Bangkok, Thailand (Online): Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Beelen%2C+K%2E%2C+Nanni%2C+F%2E%2C+Coll+Ardanuy%2C+M%2E%2C+Hosseini%2C+K%2E%2C+Tolfo%2C+G%2E%2C+%26+McGillivray%2C+B%2E+%282021%29%2E+When+time+makes+sense%3A+A+historically%2Daware+approach+to+targeted+sense+disambiguation%2E+In+Findings+of+acl%2Dij)
4. Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2016). Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606 . Coll Ardanuy, M., Nanni, F., Beelen, K., Hosseini, K., Ahnert, R., Lawrence, J., . . . McGillivray, B. (2020, December). Living machines: A study of atypical animacy. In Proceedings of the 28th international conference on computational linguistics (pp. 4534–4545). Barcelona, Spain (Online): International Committee on Computational Linguistics. Retrieved from https://www.aclweb.org/anthology/ 2020.coling-main.400 DOI: 10.18653/v1/2020.coling-main.400 [↗](https://doi.org/10.18653/v1/2020.coling-main.400) [our card](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/)
5. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019, June). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers) (pp. 4171–4186). Minneapolis, Minnesota: Association for Computational Linguistics. Retrieved from https://www.aclweb.org/anthology/N19-1423 DOI: 10.18653/v1/N19-1423 [↗](https://doi.org/10.18653/v1/n19-1423) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
6. Hämäläinen, M., & Hengchen, S. (2019). From the paft to the fiiture: a fully automatic nmt and word embeddings method for ocr post-correction. arXiv preprint arXiv:1910.05535 . Hengchen, S., Ros, R., & Marjanen, J. (2019). A data-driven approach to the changing vocabulary of the nation in english, dutch, swedish and finnish newspapers, 1750-1950. In In proceedings of the digital humanities (dh) conference. [↗](https://arxiv.org/abs/1910.05535)
7. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9 (8), 1735– 1780. [↗](https://doi.org/10.1162/neco.1997.9.8.1735)
8. Honnibal, M., Montani, I., Van Landeghem, S., & Boyd, A. (2020). spaCy: Industrial-strength Natural Language Processing in Python. Zenodo. Retrieved from https://doi.org/10.5281/zenodo .1212303 DOI: 10.5281/zenodo.1212303 [↗](https://doi.org/10.5281/zenodo)
9. Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 . Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv preprint. Retrieved 2019-11-20, from http://arxiv.org/abs/1301.3781 [↗](https://arxiv.org/abs/1412.6980) [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/)
10. Rehurek, R., & Sojka, P. (2011). Gensim–python framework for vector space modelling. NLP Centre, Faculty of Informatics, Masaryk University, Brno, Czech Republic, 3 (2). [find](https://scholar.google.com/scholar?q=Rehurek%2C+R%2E%2C+%26+Sojka%2C+P%2E+%282011%29%2E+Gensim%E2%80%93python+framework+for+vector+space+modelling%2E+NLP+Centre%2C+Faculty+of+Informatics%2C+Masaryk+University%2C+Brno%2C+Czech+Republic%2C+3+%282%29%2E)
11. Shoemark, P., Liza, F. F., Nguyen, D., Hale, S., & McGillivray, B. (2019). Room to glo: A systematic comparison of semantic change detection approaches with word embeddings. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp) (pp. 66–76). [find](https://scholar.google.com/scholar?q=Shoemark%2C+P%2E%2C+Liza%2C+F%2E+F%2E%2C+Nguyen%2C+D%2E%2C+Hale%2C+S%2E%2C+%26+McGillivray%2C+B%2E+%282019%29%2E+Room+to+glo%3A+A+systematic+comparison+of+semantic+change+detection+approaches+with+word+embeddings%2E+In+Proceedings+of+the+2019)
12. Smith, N. A. (2019). Contextual word representations: A contextual introduction. arXiv preprint arXiv:1902.06006 . van Strien, D., Beelen, K., Ardanuy, M. C., Hosseini, K., McGillivray, B., & Colavizza, G. (2020). Assessing the impact of ocr quality on downstream nlp tasks. In Icaart (1) (pp. 484–496). Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., . . . Brew, J. (2019). HuggingFace’s Transformers: State-of-the-art Natural Language Processing. ArXiv , abs/1910.03771 . [↗](https://arxiv.org/abs/1902.06006) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)

*Updated 24 October 2026*