--- title: "Evaluating Automatic Speech Recognition for Holocaust" description: "The authors test whether OpenAI's Whisper can close this gap at scale. 25% WER. 5 references." authors: ["William J.B. Mattingly", "Christy Bailey-Tomecek"] year: 2026 venue: "Proceedings of The Second Workshop on Holocaust Testimonies as Language Resources (HTRes)" source: "https://aclanthology.org/2026.htres-2.10/" doi: "10.63317/5gtr6wvyxg7y" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/mattingly-2026-evaluating-automatic-speech-recognition-holocaust/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mattingly-2026-evaluating-automatic-speech-recognition-holocaust/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/) / Evaluating Automatic Speech Recognition for Holocaust Testimonies: A Large-Scale Analysis of Whisper Performance on the Fortunoff Video Archive # Evaluating Automatic Speech Recognition for Holocaust Testimonies: A Large-Scale Analysis of Whisper Performance on the Fortunoff Video Archive **Authors:** William J.B. Mattingly and Christy Bailey-Tomecek · **Year:** 2026 · **Venue:** [Proceedings of The Second Workshop on Holocaust Testimonies as Language Resources (HTRes)](https://md.research.elamur.ai/free/en/venues/htres/) ([HTRes 2026](https://md.research.elamur.ai/free/en/venues/htres/2026/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.htres-2.10/) · [PDF](https://aclanthology.org/2026.htres-2.10.pdf) · [DOI 10.63317/5gtr6wvyxg7y](https://doi.org/10.63317/5gtr6wvyxg7y) · [OpenAlex](https://openalex.org/W7166588401) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Transcribing Holocaust survivor interviews by hand is slow and expensive. As a result, much of the material in oral-history collections remains hard to search or access. The authors test whether OpenAI's Whisper can close this gap at scale. They used 1,847 testimonies from Yale's Fortunoff Video Archive for Holocaust Testimonies, each of which already had human-corrected captions to serve as the reference. Most recordings were processed with faster-whisper and the large-v3-turbo model. Hebrew and Yiddish material went through fine-tuned ivrit-ai models instead. The outputs were scored by word error rate (WER) and the errors were examined qualitatively. Mean WER is around 15%, and roughly nine in ten testimonies fall at or below 25% WER, which the authors consider adequate for ASR-assisted human editing. Many apparent errors reflect clashing conventions rather than true misrecognitions. Whisper removes fillers, false starts and interruptions that the archive's verbatim style guide deliberately keeps. It also writes Ladino in modern Spanish spelling instead of traditional Judeo-Spanish orthography. The study supports Whisper-based transcription pipelines for testimony archives, provided they are paired with language-specific models, post-processing and expert review. ## Key points - **Data:** 1,847 Fortunoff testimonies, about 92% in English (1,699). The rest are German, French, Hebrew, Ladino, Yiddish, Slovak, ASL with English voice-over, and some with unknown language. - **Headline results:** The abstract reports a mean WER of 15.28%, with 90.9% of testimonies at WER ≤25% ("Fair" or better). The conclusion gives a slightly different figure (16.3%, framed as 11 languages), so the numbers are not fully consistent within the paper. - **Setup:** faster-whisper 1.0.0 with openai/whisper-large-v3-turbo, plus ivrit-ai/whisper-large-v3-turbo-ct2 (Hebrew) and ivrit-ai/yi-whisper-large-v3-turbo-ct2 (Yiddish). Text processing used spaCy 3.7 and scoring used a custom difflib-based evaluation. - **Error sources:** These include disfluencies, interrupted speech, elderly and emotional speakers, code-switching, degraded tape audio, and rare place names and camp jargon. The reference captions are themselves imperfect, with timing misalignment and editor inconsistencies. - **Recommended workflow and future work:** The proposed workflow is language detection, then language-specific model selection, then post-processing to restore disfluencies and heritage orthography. Future work includes fine-tuning on testimonies, building a first Ladino ASR model, and Holocaust-specific NER. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Yale University](https://md.research.elamur.ai/free/en/universities/yale-university/) — University, New Haven, United States · 95 works · 12,765 citations 📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 452 works 🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 480 works ## Cite Cite as: William J.B. Mattingly and Christy Bailey-Tomecek. 2026. Evaluating Automatic Speech Recognition for Holocaust Testimonies: A Large-Scale Analysis of Whisper Performance on the Fortunoff Video Archive. Proceedings of The Second Workshop on Holocaust Testimonies as Language Resources (HTRes). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{mattingly2026evaluating, title = {{Evaluating Automatic Speech Recognition for Holocaust Testimonies: A Large-Scale Analysis of Whisper Performance on the Fortunoff Video Archive}}, author = {William J.B. Mattingly and Christy Bailey-Tomecek}, booktitle = {Proceedings of The Second Workshop on Holocaust Testimonies as Language Resources (HTRes)}, year = {2026}, doi = {10.63317/5gtr6wvyxg7y}, url = {https://doi.org/10.63317/5gtr6wvyxg7y} } ``` - [← previous Evaluating Automated Scoring Models on Official ENEM Essays](https://md.research.elamur.ai/free/en/papers/rossman-2026-evaluating-automated-scoring-models-official/) - [next → Evaluating Communicative Belief Updates in Large Language Models via Implicature Recognition and Cancellation](https://md.research.elamur.ai/free/en/papers/piano-2026-evaluating-communicative-belief-updates-large/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/) ## References 5 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Bailey-Tomecek, C. (2025, October 28). Comparing Sonix and aTrain for transcribing French, German, Hebrew, and Russian Testimonies [Presentation] AI for Libraries, Archives, and Museums Speech-to-Text Working Group. [find](https://scholar.google.com/scholar?q=Bailey%2DTomecek%2C+C%2E+%282025%2C+October+28%29%2E+Comparing+Sonix+and+aTrain+for+transcribing+French%2C+German%2C+Hebrew%2C+and+Russian+Testimonies+%5BPresentation%5D+AI+for+Libraries%2C+Archives%2C+and+Museums+Speech%2Dto%2DText) 2. Rao, N., O’Riodan, S., & Coulis, J. (2025). AI and labor: Captioning audiovisual content with Whisper. IFLA Journal, 51(3), 803-813. https://doi.org/10.1177/03400352241310534 — [doi:10.1177/03400352241310534](https://doi.org/10.1177/03400352241310534) [↗](https://doi.org/10.1177/03400352241310534) 3. Rodriguez, D., & Brown, B.J. (2023). Comparative analysis of automated speech recognition technologies for enhanced audiovisual accessibility. Code4Lib Journal 58. [https://journal.code4lib.org/articles/17820](https://journal.code4lib.org/articles/17820) [↗](https://journal.code4lib.org/articles/17820) 4. Dunn, J., Lynema, E., Wheeler, B., Cameron, J. (2024, October 18). Whisper applied to digitized historical audiovisual materials [Conference presentation]. [find](https://scholar.google.com/scholar?q=Dunn%2C+J%2E%2C+Lynema%2C+E%2E%2C+Wheeler%2C+B%2E%2C+Cameron%2C+J%2E+%282024%2C+October+18%29%2E+Whisper+applied+to+digitized+historical+audiovisual+materials+%5BConference+presentation%5D%2E) 5. Fantastic Futures 2024. [find](https://scholar.google.com/scholar?q=Fantastic+Futures+2024%2E) *Updated 1 November 2026*