---
title: "Bootstrapping AI — paper by Backer and Hyman (2025)"
description: "And machine-made transcriptions could replace much crowdsourced labor for institutions with limited… 30 references, cited by 2 papers here."
authors: ["Samuel Backer", "Louis Hyman"]
year: 2025
venue: "Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities"
source: "https://aclanthology.org/2025.nlp4dh-1.21/"
doi: "10.18653/v1/2025.nlp4dh-1.21"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/backer-2025-bootstrapping-ai-interdisciplinary-approaches-assessing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/backer-2025-bootstrapping-ai-interdisciplinary-approaches-assessing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/) / Bootstrapping AI: Interdisciplinary Approaches to Assessing OCR Quality in English-Language Historical Documents

# Bootstrapping AI: Interdisciplinary Approaches to Assessing OCR Quality in English-Language Historical Documents

**Authors:** Samuel Backer and Louis Hyman · **Year:** 2025 · **Venue:** [Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities](https://md.research.elamur.ai/free/en/venues/nlp4dh/) ([International Conference on Natural Language Processing for the Digital Humanities 2025](https://md.research.elamur.ai/free/en/venues/nlp4dh/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.nlp4dh-1.21/) · [PDF](https://aclanthology.org/2025.nlp4dh-1.21.pdf) · [DOI 10.18653/v1/2025.nlp4dh-1.21](https://doi.org/10.18653/v1/2025.nlp4dh-1.21) · [OpenAlex](https://openalex.org/W4411120084)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/backer-2025-bootstrapping-ai-interdisciplinary-approaches-assessing/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Backer and Hyman ask how to judge LLM-assisted OCR for historians, and whether cheaper correction pipelines change research outcomes. Their test bed is the Samuel Gompers correspondence at the Library of Congress: clean scans of faded letter-press copies, about 20,000 of which volunteers have transcribed. They compare several systems against these human transcriptions:

- Tesseract as the baseline;
- vision-to-text LLaMA 3.2 and ChatGPT-4o;
- text-only ChatGPT-4o used to correct Tesseract output;
- local BART and ByT5 correctors, fine-tuned on either human ("gold") or LLM-produced ("silver") targets.

Vision models win on every measure. By CER/WER, the fine-tuned correctors look worse than raw Tesseract. By precision, NER results and BERT embedding similarity, however, they clearly beat Tesseract and come close to the flagship models. Silver training data works almost as well as gold, and a large silver set outperforms a smaller gold one.

The authors draw two conclusions. Edit-distance metrics can hide gains that matter for humanities work, so evaluation should follow the task. And machine-made transcriptions could replace much crowdsourced labor for institutions with limited budgets.

## Key points
- **Data:** the Gompers collection holds about 500,000 letters. The authors used 10,000 crowd-transcribed letters (from the "We The People" project) for training and 1,000 for testing; letters average 134.4 words.
- **Text-only ChatGPT-4o correction:**
  - WER fell 42% on average and 68% at the median versus Tesseract.
  - CER fell 15% on average and 41% at the median.
  - This contradicts earlier pessimistic findings that used ChatGPT-4.
- **BART and ByT5 correctors by CER/WER:**
  - Trained on 100, 1,000 and 10,000 pairs, they mostly scored below Tesseract.
  - Only the median WER of the 10K models improved.
  - The best ByT5 models still had error rates above 80%.
  - At 10K examples, gold- and silver-trained models were within 3% on median CER and WER.
- **Precision:**
  - The best corrector, BART trained on 10K LLaMA transcriptions, reached 92.6, against 95.6 for ChatGPT-4o and 96.5 for LLaMA.
  - The best fine-tuned models improved precision by more than 30% over Tesseract.
- **Downstream tasks:**
  - With spaCy NER, the vision models beat Tesseract by 157%, with a 14% gap to the best fine-tuned model.
  - ByT5 slightly beat BART on NER, possibly because BART tends to hallucinate.
  - On BERT cosine similarity, the top trained models were 5% above Tesseract and within 0.5% of a flagship multimodal model.

## Interesting ideas
- Strong but costly models can label a large training set so that a small local model inherits much of their skill; more silver data outweighed less gold data.
- Pick the metric by which error is costlier: historians accept gaps but not invented content, so precision fits them better than edit distance.
- An upstream change can look harmful on surface metrics yet help downstream, so it should also be judged by its effect on later stages.
- A model that rewrites fluently can score well overall while hallucinating, so a fabrication-sensitive check should sit beside aggregate scores.
- Cosine similarity of sentence embeddings between gold and system output was used as a cheap proxy for whether meaning survived the edits.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): when witness parsers or an LLM teacher produce silver trees, quantity can partly compensate for imperfect labels, which justifies scaling silver sets before buying more gold annotation.
- **I2** (world): score extracted entities, states and holdings mainly on whether asserted facts are true, and penalize fabricated facts more heavily than missed ones.
- **I3** (tokens): evaluate normalization or tokenizer changes on noisy input by whether the resulting UD trees and lemmas match gold, not only by character-level differences.
- **I4** (back): test the text → graph → text compressor by checking that each regenerated statement is supported by the source graph, not just by how similar the surface text looks.
- **I5** (learn): add a meaning-preservation check as an extra signal in evaluation runs, alongside negative controls, to see when surface errors do not change content.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Johns Hopkins University](https://md.research.elamur.ai/free/en/universities/johns-hopkins-university/) — University, Baltimore, United States · 247 works · 106,235 citations
- [University of Maine](https://md.research.elamur.ai/free/en/universities/university-of-maine/) — University, Orono, United States · 2 works · 11 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [📚 Literature and Literary Theory](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/literature-and-literary-theory/) · 541 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,379 works

## Cite

Cite as: Samuel Backer and Louis Hyman. 2025. Bootstrapping AI: Interdisciplinary Approaches to Assessing OCR Quality in English-Language Historical Documents. Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{backer2025bootstrapping,
  title = {{Bootstrapping AI: Interdisciplinary Approaches to Assessing OCR Quality in English-Language Historical Documents}},
  author = {Samuel Backer and Louis Hyman},
  booktitle = {Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities},
  year = {2025},
  doi = {10.18653/v1/2025.nlp4dh-1.21},
  url = {https://doi.org/10.18653/v1/2025.nlp4dh-1.21}
}
```

- [← previous Beyond Base Predictors: Using LLMs to Resolve Ambiguities in Akkadian Lemmatization](https://md.research.elamur.ai/free/en/papers/riemenschneider-2025-beyond-base-predictors-llms-resolve/)
- [next → Breaking Bad: Norms for Valence, Arousal, and Dominance for over 10k English Multiword Expressions](https://md.research.elamur.ai/free/en/papers/mohammad-2025-breaking-bad-norms-valence-arousal/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/)

## References

30 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774. [↗](https://arxiv.org/abs/2303.08774)
2. Kunga Bang. 2024. Exploring generative large language models for post-ocr enhancement of historical texts. [find](https://scholar.google.com/scholar?q=Kunga+Bang%2E+2024%2E+Exploring+generative+large+language+models+for+post%2Docr+enhancement+of+historical+texts%2E)
3. Angel Beshirov, Milena Dobreva, Dimitar Dimitrov, Momchil Hardalov, Ivan Koychev, and Preslav Nakov. 2024. Post-ocr text correction for bulgarian historical documents. arXiv preprint arXiv:2409.00527. [↗](https://arxiv.org/abs/2409.00527)
4. Callum William Booth, Alan Thomas, and Robert Gaizauskas. 2024. Bln600: A parallel corpus of machine/human transcribed nineteenth century newspaper texts. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 2440–2446. [find](https://scholar.google.com/scholar?q=Callum+William+Booth%2C+Alan+Thomas%2C+and+Robert+Gaizauskas%2E+2024%2E+Bln600%3A+A+parallel+corpus+of+machine%2Fhuman+transcribed+nineteenth+century+newspaper+texts%2E+In+Proceedings+of+the+2024+Joint+Internationa)
5. Emanuela Boros, Maud Ehrmann, Matteo Romanello, Sven Najem-Meyer, and Frédéric Kaplan. 2024. Post-correction of historical text transcripts with large language models: An exploratory study. LaTeCH-CLfL 2024, pages 133–159. [find](https://scholar.google.com/scholar?q=Emanuela+Boros%2C+Maud+Ehrmann%2C+Matteo+Romanello%2C+Sven+Najem%2DMeyer%2C+and+Fr%C3%A9d%C3%A9ric+Kaplan%2E+2024%2E+Post%2Dcorrection+of+historical+text+transcripts+with+large+language+models%3A+An+exploratory+study%2E+LaTeCH%2DCLf)
6. Jonathan Bourne. 2024. Clocr-c: Context leveraging ocr correction with pre-trained language models. arXiv preprint arXiv:2408.17428. [↗](https://arxiv.org/abs/2408.17428)
7. Ryan Cordell. 2017. " q i-jtb the raven": Taking dirty ocr seriously. Book History, 20(1):188–225. [find](https://scholar.google.com/scholar?q=Ryan+Cordell%2E+2017%2E+%22+q+i%2Djtb+the+raven%22%3A+Taking+dirty+ocr+seriously%2E+Book+History%2C+20%281%29%3A188%E2%80%93225%2E)
8. Florian Debaene, Aaron Maladry, Els Lefever, and Veronique Hoste. 2025. Evaluating transformers for ocr post-correction in early modern dutch theatre. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10367–10374. [find](https://scholar.google.com/scholar?q=Florian+Debaene%2C+Aaron+Maladry%2C+Els+Lefever%2C+and+Veronique+Hoste%2E+2025%2E+Evaluating+transformers+for+ocr+post%2Dcorrection+in+early+modern+dutch+theatre%2E+In+Proceedings+of+the+31st+International+Conferen)
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186. [↗](https://doi.org/10.18653/v1/n19-1423)
10. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783. [↗](https://arxiv.org/abs/2407.21783)
11. Alex Ghiriti, Wolfgang Göderle, and Roman Kern. 2024. Exploring the capabilities of gpt4-vision as ocr engine. In International Conference on Theory and Practice of Digital Libraries, pages 3–12. Springer. [find](https://scholar.google.com/scholar?q=Alex+Ghiriti%2C+Wolfgang+G%C3%B6derle%2C+and+Roman+Kern%2E+2024%2E+Exploring+the+capabilities+of+gpt4%2Dvision+as+ocr+engine%2E+In+International+Conference+on+Theory+and+Practice+of+Digital+Libraries%2C+pages+3%E2%80%9312%2E+Spri)
12. Jo Guldi. 2023. The dangerous art of text mining: A methodology for digital history. Cambridge University Press. [↗](https://doi.org/10.1017/9781009263016)
13. Arthur Hemmer, Mickaël Coustaty, Nicola Bartolo, and Jean-Marc Ogier. 2024. Confidence-aware document ocr error detection. In International Workshop on Document Analysis Systems, pages 213–228. Springer. [find](https://scholar.google.com/scholar?q=Arthur+Hemmer%2C+Micka%C3%ABl+Coustaty%2C+Nicola+Bartolo%2C+and+Jean%2DMarc+Ogier%2E+2024%2E+Confidence%2Daware+document+ocr+error+detection%2E+In+International+Workshop+on+Document+Analysis+Systems%2C+pages+213%E2%80%93228%2E+Spring)
14. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear. [find](https://scholar.google.com/scholar?q=Matthew+Honnibal+and+Ines+Montani%2E+2017%2E+spaCy+2%3A+Natural+language+understanding+with+Bloom+embeddings%2C+convolutional+neural+networks+and+incremental+parsing%2E+To+appear%2E)
15. Mark Humphries. 2023. History and generative ai. Teaching History, 57(3):4–9. [find](https://scholar.google.com/scholar?q=Mark+Humphries%2E+2023%2E+History+and+generative+ai%2E+Teaching+History%2C+57%283%29%3A4%E2%80%939%2E)
16. Mark Humphries, Lianne C Leddy, Quinn Downton, Meredith Legace, John McConnell, Isabella Murray, and Elizabeth Spence. 2024. Unlocking the archives: Using large language models to transcribe handwritten historical documents. arXiv preprint arXiv:2411.03340. [↗](https://arxiv.org/abs/2411.03340)
17. Jenna Kanerva, Cassandra Ledins, Siiri Käpyaho, and Filip Ginter. 2025. Ocr error post-correction with llms in historical documents: No free lunches. arXiv preprint arXiv:2502.01205. [↗](https://arxiv.org/abs/2502.01205)
18. Seorin Kim, Julien Baudru, Wouter Ryckbosch, Hugues Bersini, and Vincent Ginis. 2025. Early evidence of how llms outperform traditional systems on ocr/htr tasks for historical records. arXiv preprint arXiv:2501.11623. [↗](https://arxiv.org/abs/2501.11623)
19. Mike Lewis. 2019. [Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). arXiv preprint arXiv:1910.13461. [↗](https://arxiv.org/abs/1910.13461)
20. Lucian Li. 2024. Handwriting recognition in historical documents with multimodal llm. arXiv preprint arXiv:2410.24034. Library of Congress, Manuscript Division. n.d. American Federation of Labor Records. https://crowd.loc.gov/campaigns/afl/. [↗](https://arxiv.org/abs/2410.24034)
21. Laura Manrique-Gómez, Tony Montes, Arturo Rodríguez-Herrera, and Rubén Manrique. 2024. Historical ink: 19th century latin american spanish newspaper corpus with llm ocr correction. arXiv preprint arXiv:2407.12838. [↗](https://arxiv.org/abs/2407.12838)
22. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. [Transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38–45.
23. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. Byt5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306. [↗](https://doi.org/10.1162/tacl_a_00461)
24. James Zhang, Wouter Haverals, Mary Naydan, and Brian W Kernighan. 2024. Post-ocr correction with openai’s gpt models on challenging english prosody texts. In Proceedings of the ACM Symposium on Document Engineering 2024, pages 1–4. [find](https://scholar.google.com/scholar?q=James+Zhang%2C+Wouter+Haverals%2C+Mary+Naydan%2C+and+Brian+W+Kernighan%2E+2024%2E+Post%2Docr+correction+with+openai%E2%80%99s+gpt+models+on+challenging+english+prosody+texts%2E+In+Proceedings+of+the+ACM+Symposium+on+Docume)
25. Clemens Neudecker, Konstantin Baierer, Mike Gerber, Christian Clausner, Apostolos Antonacopoulos, and Stefan Pletschacher. 2021. A survey of ocr evaluation tools and metrics. In Proceedings of the 6th International Workshop on Historical Document Imaging and Processing, pages 13–18. [find](https://scholar.google.com/scholar?q=Clemens+Neudecker%2C+Konstantin+Baierer%2C+Mike+Gerber%2C+Christian+Clausner%2C+Apostolos+Antonacopoulos%2C+and+Stefan+Pletschacher%2E+2021%2E+A+survey+of+ocr+evaluation+tools+and+metrics%2E+In+Proceedings+of+the+6th)
26. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830. [find](https://scholar.google.com/scholar?q=Fabian+Pedregosa%2C+Ga%C3%ABl+Varoquaux%2C+Alexandre+Gramfort%2C+Vincent+Michel%2C+Bertrand+Thirion%2C+Olivier+Grisel%2C+Mathieu+Blondel%2C+Peter+Prettenhofer%2C+Ron+Weiss%2C+Vincent+Dubourg%2C+et+al%2E+2011%2E+Scikit%2Dlearn%3A+Mach)
27. Christophe Rigaud, Antoine Doucet, Mickaël Coustaty, and Jean-Philippe Moreux. 2019. Icdar 2019 competition on post-ocr text correction. In 2019 international conference on document analysis and recognition (ICDAR), pages 1588–1593. IEEE. [find](https://scholar.google.com/scholar?q=Christophe+Rigaud%2C+Antoine+Doucet%2C+Micka%C3%ABl+Coustaty%2C+and+Jean%2DPhilippe+Moreux%2E+2019%2E+Icdar+2019+competition+on+post%2Docr+text+correction%2E+In+2019+international+conference+on+document+analysis+and+recog)
28. Ray Smith. 2007. An overview of the tesseract ocr engine. In Ninth international conference on document analysis and recognition (ICDAR 2007), volume 2, pages 629–633. IEEE. [↗](https://doi.org/10.1109/icdar.2007.4376991)
29. Myriam C Traub, Jacco Van Ossenbruggen, and Lynda Hardman. 2015. Impact analysis of ocr quality on research tasks in digital archives. In Research and Advanced Technology for Digital Libraries: 19th International Conference on Theory and Practice of Digital Libraries, TPDL 2015, Poznań, Poland, September 14-18, 2015, Proceedings 19, pages 252– 263. Springer. [find](https://scholar.google.com/scholar?q=Myriam+C+Traub%2C+Jacco+Van+Ossenbruggen%2C+and+Lynda+Hardman%2E+2015%2E+Impact+analysis+of+ocr+quality+on+research+tasks+in+digital+archives%2E+In+Research+and+Advanced+Technology+for+Digital+Libraries%3A+19th+I)
30. Daniel van Strien, Kaspar Beelen, Mariona Coll Ardanuy, Kasra Hosseini, Barbara McGillivray, and Giovanni Colavizza. 2020. Assessing the impact of ocr quality on downstream nlp tasks. In Proceedings of the 12th International Conference on Agents and Artificial Intelligence - Volume 1: ARTIDIGH, pages 484–496. INSTICC, SciTePress. [find](https://scholar.google.com/scholar?q=Daniel+van+Strien%2C+Kaspar+Beelen%2C+Mariona+Coll+Ardanuy%2C+Kasra+Hosseini%2C+Barbara+McGillivray%2C+and+Giovanni+Colavizza%2E+2020%2E+Assessing+the+impact+of+ocr+quality+on+downstream+nlp+tasks%2E+In+Proceedings+o)

*Updated 1 November 2026*