---
title: "Large-scale Language Model Rescoring on Long-form Data (2023)"
description: "The authors present this as the first study to scale LLMs to long-form ASR. 19 references, cited by 3 papers here."
authors: ["Tongzhou Chen", "Cyril Allauzen", "Yinghui Huang", "Daniel Park", "David Rybach", "W. Ronny Huang", "Rodrigo Cabrera", "Kartik Audhkhasi", "Bhuvana Ramabhadran", "Pedro J. Moreno", "Michael Riley"]
year: 2023
venue: "arXiv:2306.08133 · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)"
source: "https://arxiv.org/abs/2306.08133"
doi: "10.48550/arxiv.2306.08133"
arxiv: "2306.08133"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2023-large-scale-language-model-rescoring/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2023-large-scale-language-model-rescoring/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/) / Large-scale Language Model Rescoring on Long-form Data

# Large-scale Language Model Rescoring on Long-form Data

**Authors:** Tongzhou Chen, Cyril Allauzen, [Yinghui Huang](https://md.research.elamur.ai/free/en/authors/yinghui-huang/), [Daniel Park](https://md.research.elamur.ai/free/en/authors/daniel-park/), David Rybach, W. Ronny Huang, Rodrigo Cabrera, [Kartik Audhkhasi](https://md.research.elamur.ai/free/en/authors/kartik-audhkhasi/), [Bhuvana Ramabhadran](https://md.research.elamur.ai/free/en/authors/bhuvana-ramabhadran/), [Pedro J. Moreno](https://md.research.elamur.ai/free/en/authors/pedro-j-moreno/) and Michael Riley · **Year:** 2023 · **Venue:** [arXiv:2306.08133 · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)](https://md.research.elamur.ai/free/en/venues/ieee-international-conference-on-acoustics-speech-and-signal-processing/) ([IEEE International Conference on Acoustics Speech and Signal Processing 2023](https://md.research.elamur.ai/free/en/venues/ieee-international-conference-on-acoustics-speech-and-signal-processing/2023/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2306.08133) · [PDF](https://arxiv.org/pdf/2306.08133) · [DOI 10.48550/arxiv.2306.08133](https://doi.org/10.48550/arxiv.2306.08133) · [arXiv 2306.08133](https://arxiv.org/abs/2306.08133) · [OpenAlex](https://openalex.org/W4372269772)

**Citations:** 18 citations (OpenAlex, 2026-10-14) · 3 in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether very large language models help speech recognition of long recordings, using 30-minute YouTube videos as the test material. The first-pass recognizer is a conformer with HAT factorization. When a segment ends, its lattice is rescored by an external LM. The internal-LM estimate is subtracted and the external score is added, each with its own tuned weight. The external LMs compared are T5 and MT5 (XL, XXL), PaLM (8B–540B), a 70M conformer LM and a 4.5B MaxEnt LM. For code-switched Indian English, MT5 was further fine-tuned on WEBDOC, a Hindi/Indian-English web and Wikipedia corpus transliterated to Latin script. Two engineering choices matter a lot. First, a decoder with limited label context merges hypotheses, so the lattice becomes a real graph instead of a tree. Second, the previous segments' 1-best output is fed to the scorer as context. Against a MaxEnt first-pass baseline, WER falls by up to 8% relative and Salient Term Error Rate by up to 30% relative. Gains from LLMs and from a conventional neural LM add up. The authors present this as the first study to scale LLMs to long-form ASR.

## Key points
- Test locales: en-us and en-in (code-switched with Hindi), built from long-form YouTube audio. A 5% slice of the test utterances served as a dev set for tuning hyperparameters.
- Combined score: S_HAT(Y|X) − µ·S_ILM(Y) + ν·S_ELM(Y), where µ and ν are tunable weights.
- T5 and MT5 were pre-trained with span corruption for 1M steps, then fine-tuned for 100K steps on a prefix-LM objective. MT5 got 300K more LM steps on WEBDOC (170M sentences, 2.9B tokens). PaLM was used as-is.
- The first-pass decoder conditions only on the two previous word-pieces. Beam hypotheses that share this context collapse into one lattice state.
- Results: up to 8% relative WER reduction and up to 30% relative STER reduction versus a MaxEnt first-pass baseline. Combining the LLM and the conformer LM gives additive gains.

## Interesting ideas
- When adding an external scorer, subtract the model's own internal prior so that the same preference is not counted twice.
- Bounding the history a decoder conditions on makes hypotheses with the same recent context merge, so a fixed beam yields a graph that holds many more alternatives than a trie.
- Score each segment with the already-decided best output of the previous segments as context, rather than scoring segments in isolation.
- Besides the overall error rate, track errors only on salient terms, since those carry the meaning users care about.
- A huge general model and a small in-domain model can give complementary, additive improvements when interpolated.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tree): When gparse n-best parses are rescored with an external scorer, discount the parser's built-in bias toward frequent attachments before mixing in the external score.
- **I2** (tokens): The tagger/morphology beam could merge states that share their last two tags, giving a compact lattice of readings that later stages can rescore.
- **I3** (world): When interpreting a new sentence, condition its ambiguity resolution on the committed world state from earlier sentences, keeping each decision traceable to that context.
- **I4** (learn): Alongside LAS, report accuracy on content-bearing predicates and entities, which reveals whether gains actually reach frames and answers.
- **I5** (learn): A broad LLM teacher tier and a small domain-trained witness parser could be combined as separate weighted votes when building silver data, rather than picking only one.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 430 works · 424,363 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 452 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Tongzhou Chen, Cyril Allauzen, Yinghui Huang, Daniel Park, David Rybach, W. Ronny Huang, Rodrigo Cabrera, Kartik Audhkhasi, Bhuvana Ramabhadran, Pedro J. Moreno and Michael Riley. 2023. Large-scale Language Model Rescoring on Long-form Data. arXiv:2306.08133 · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chen2023large,
  title = {{Large-scale Language Model Rescoring on Long-form Data}},
  author = {Tongzhou Chen, Cyril Allauzen, Yinghui Huang, Daniel Park, David Rybach, W. Ronny Huang, Rodrigo Cabrera, Kartik Audhkhasi, Bhuvana Ramabhadran, Pedro J. Moreno and Michael Riley},
  year = {2023},
  doi = {10.48550/arxiv.2306.08133},
  eprint = {2306.08133},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.08133}
}
```

- [← previous Knowledge Sanitization of Large Language Models](https://md.research.elamur.ai/free/en/papers/ishibashi-2023-knowledge-sanitization-large-language-models/)
- [next → ModuleFormer: Modularity Emerges from Mixture-of-Experts](https://md.research.elamur.ai/free/en/papers/shen-2023-moduleformer-modularity-emerges-mixture-experts/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/)

## References

19 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. C. Raffel et al., “[Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/).” J. Mach. Learn. Res., vol. 21, no. 140, pp. 1–67, 2020. [↗](https://doi.org/10.48550/arxiv.1910.10683)
2. A. Wang and K. Cho, “[Bert has a mouth, and it must speak: Bert as a markov random field language model](https://md.research.elamur.ai/free/en/papers/wang-2019-bert-has-mouth-must-speak/),” arXiv preprint arXiv:1902.04094, 2019. [20] C. Allauzen, E. Variani, M. Riley, D. Rybach, and H. Zhang, “A hybrid seq-2-seq ASR design for on-device and server applications,” in Interspeech 2021, 2021, pp. 4044–4048. [21] T. N. Sainath et al., “An efficient streaming non-recurrent on-device end-to-end model with improvements to rare-word modeling,” in Interspeech, 2021, pp. 1777–1781. [22] K. Hu et al., “Improving deliberation by text-only and semi-supervised training,” arXiv preprint arXiv:2206.14716, 2022. [↗](https://arxiv.org/abs/1902.04094)
3. C. Raffel et al., “[Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/),” Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020. [23] L. Xue et al., “mT5: A massively multilingual pre-trained text-to-text transformer,” in 2021 NAACL: Human Language Technologies, Jun. 2021.
4. T. Brown et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020. [24] A. Gulati et al., “[Conformer: Convolution-augmented transformer for speech recognition](https://md.research.elamur.ai/free/en/papers/gulati-2020-conformer-convolution-augmented-transformer-speech/),” arXiv preprint arXiv:2005.08100, 2020. [↗](https://arxiv.org/abs/2005.08100)
5. A. Chowdhery et al., “[Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/),” arXiv preprint arXiv:2204.02311, 2022. [25] H. Liao, E. McDermott, and A. Senior, “Large scale deep neural network acoustic modeling with semi-supervised training data for youtube video transcription,” in ASRU, 2013, pp. 368– 373. [↗](https://arxiv.org/abs/2204.02311)
6. X. Zheng, C. Zhang, and P. C. Woodland, “Adapting gpt, gpt-2 and bert language models for speech recognition,” in 2021 IEEE ASRU, 2021, pp. 162–168. [find](https://scholar.google.com/scholar?q=X%2E+Zheng%2C+C%2E+Zhang%2C+and+P%2E+C%2E+Woodland%2C+%E2%80%9CAdapting+gpt%2C+gpt%2D2+and+bert+language+models+for+speech+recognition%2C%E2%80%9D+in+2021+IEEE+ASRU%2C+2021%2C+pp%2E+162%E2%80%93168%2E)
7. J. Salazar, D. Liang, T. Q. Nguyen, and K. Kirchhoff, “[Masked language model scoring](https://md.research.elamur.ai/free/en/papers/salazar-2020-masked-language-model-scoring/),” in 2020 ACL, Jul. 2020.
8. E. Variani et al., “Neural oracle search on n-best hypotheses,” in ICASSP, 2020, pp. 7824–7828. [find](https://scholar.google.com/scholar?q=E%2E+Variani+et+al%2E%2C+%E2%80%9CNeural+oracle+search+on+n%2Dbest+hypotheses%2C%E2%80%9D+in+ICASSP%2C+2020%2C+pp%2E+7824%E2%80%937828%2E)
9. S.-H. Chiu and B. Chen, “Innovative bert-based reranking language models for speech recognition,” in SLT. IEEE, 2021, pp. 266–271. [find](https://scholar.google.com/scholar?q=S%2E%2DH%2E+Chiu+and+B%2E+Chen%2C+%E2%80%9CInnovative+bert%2Dbased+reranking+language+models+for+speech+recognition%2C%E2%80%9D+in+SLT%2E+IEEE%2C+2021%2C+pp%2E+266%E2%80%93271%2E)
10. T. Hori, Y. Kubo, and A. Nakamura, “Real-time one-pass decoding with recurrent neural network language model for speech recognition,” in ICASSP. IEEE, 2014, pp. 6364–6368. [find](https://scholar.google.com/scholar?q=T%2E+Hori%2C+Y%2E+Kubo%2C+and+A%2E+Nakamura%2C+%E2%80%9CReal%2Dtime+one%2Dpass+decoding+with+recurrent+neural+network+language+model+for+speech+recognition%2C%E2%80%9D+in+ICASSP%2E+IEEE%2C+2014%2C+pp%2E+6364%E2%80%936368%2E)
11. J. Chorowski and N. Jaitly, “Towards better decoding and language model integration in sequence to sequence models,” arXiv preprint arXiv:1612.02695, 2016. [↗](https://arxiv.org/abs/1612.02695)
12. T. Hori, J. Cho, and S. Watanabe, “End-to-end speech recognition with word-based rnn language models,” in SLT. IEEE, 2018, pp. 389–396. [find](https://scholar.google.com/scholar?q=T%2E+Hori%2C+J%2E+Cho%2C+and+S%2E+Watanabe%2C+%E2%80%9CEnd%2Dto%2Dend+speech+recognition+with+word%2Dbased+rnn+language+models%2C%E2%80%9D+in+SLT%2E+IEEE%2C+2018%2C+pp%2E+389%E2%80%93396%2E)
13. C. Peyser et al., “Improving tail performance of a deliberation e2e asr model using a large text corpus,” arXiv preprint arXiv:2008.10491, 2020. [↗](https://arxiv.org/abs/2008.10491)
14. A. Sriram, H. Jun, S. Satheesh, and A. Coates, “Cold fusion: Training seq2seq models together with language models,” arXiv preprint arXiv:1708.06426, 2017. [↗](https://arxiv.org/abs/1708.06426)
15. C. Gulcehre et al., “[On using monolingual corpora in neural machine translation](https://md.research.elamur.ai/free/en/papers/gulcehre-2015-using-monolingual-corpora-neural-machine/),” arXiv preprint arXiv:1503.03535, 2015. [↗](https://arxiv.org/abs/1503.03535)
16. C. Shan et al., “Component fusion: Learning replaceable language model component for end-to-end speech recognition system,” in ICASSP. IEEE, 2019, pp. 5361–5635. [find](https://scholar.google.com/scholar?q=C%2E+Shan+et+al%2E%2C+%E2%80%9CComponent+fusion%3A+Learning+replaceable+language+model+component+for+end%2Dto%2Dend+speech+recognition+system%2C%E2%80%9D+in+ICASSP%2E+IEEE%2C+2019%2C+pp%2E+5361%E2%80%935635%2E)
17. A. Kannan et al., “An analysis of incorporating an external language model into a sequence-to-sequence model,” in ICASSP, 2018, pp. 1–5828. [find](https://scholar.google.com/scholar?q=A%2E+Kannan+et+al%2E%2C+%E2%80%9CAn+analysis+of+incorporating+an+external+language+model+into+a+sequence%2Dto%2Dsequence+model%2C%E2%80%9D+in+ICASSP%2C+2018%2C+pp%2E+1%E2%80%935828%2E)
18. E. Variani, D. Rybach, C. Allauzen, and M. Riley, “Hybrid autoregressive transducer (hat),” in ICASSP, 2020, pp. 6139– 6143. [find](https://scholar.google.com/scholar?q=E%2E+Variani%2C+D%2E+Rybach%2C+C%2E+Allauzen%2C+and+M%2E+Riley%2C+%E2%80%9CHybrid+autoregressive+transducer+%28hat%29%2C%E2%80%9D+in+ICASSP%2C+2020%2C+pp%2E+6139%E2%80%93+6143%2E)
19. E. McDermott, H. Sak, and E. Variani, “A density ratio approach to language model fusion in end-to-end automatic speech recognition,” in ASRU, 2019, pp. 434–441. [26] A. Narayanan et al., “Recognizing long-form speech using streaming end-to-end models,” in ASRU, 2019, pp. 920–927. [27] J. Emond, B. Ramabhadran, B. Roark, P. Moreno, and M. Ma, “Transliteration based approaches to improve code-switched speech recognition performance,” in SLT. IEEE, 2018, pp. 448–455. [28] G. Wenzek et al., “[Ccnet: Extracting high quality monolingual datasets from web crawl data](https://md.research.elamur.ai/free/en/papers/wenzek-2020-ccnet-extracting-high-quality-monolingual/),” arXiv preprint arXiv:1911.00359, 2019. [29] R. Botros et al., “Tied & reduced rnn-t decoder,” arXiv preprint arXiv:2109.07513, 2021. [30] F. Biadsy, K. Hall, P. Moreno, and B. Roark, “Backoff inspired features for maximum entropy language models,” 2014. [31] F. Biadsy, M. Ghodsi, and D. Caseiro, “Effectively building tera scale maxent language models incorporating non-linguistic signals,” 2017. [32] A. Tripathi, H. Lu, H. Sak, and H. Soltau, “Monotonic recurrent neural network transducer and decoding strategies,” in ASRU, 2019, pp. 944—-948. [33] R. Zazo, T. N. Sainath, G. Simko, and C. Parada, “Feature learning with raw-waveform cldnns for voice activity detection,” in Interspeech, 2016, pp. 3668–3672. [34] R. Prabhavalkar et al., “Less is more: Improved rnn-t decoding using limited label context and path merging,” in ICASSP, 2021, pp. 5659–5663. [35] J. Kaplan et al., “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361, 2020. [36] J. Hoffmann et al., “Training compute-optimal large language models,” arXiv preprint arXiv:2203.15556, 2022. [37] V. Ravi et al., “Improving accuracy of rare words for rnn-transducer through unigram shallow fusion,” arXiv preprint arXiv:2012.00133, 2020. [↗](https://arxiv.org/abs/1911.00359)

*Updated 1 November 2026*