---
title: "Reward-Guided Fine-Tuning of Whisper for Low-Resource Nepali"
description: "The approach matters because it brings human quality judgments into training without full RLHF, and… 17 references, cited by 1 paper here."
authors: ["Aadarsh Pandit", "Yudhin Khanal", "Ishan Pandey", "Kushal Kunwar", "Sunil Regmi"]
year: 2026
venue: "Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)"
source: "https://aclanthology.org/2026.chipsal-1.16/"
doi: "10.63317/45u9hjmchyhh"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pandit-2026-reward-guided-fine-tuning-whisper/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pandit-2026-reward-guided-fine-tuning-whisper/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/16/) / Reward-Guided Fine-Tuning of Whisper for Low-Resource Nepali Speech Recognition

# Reward-Guided Fine-Tuning of Whisper for Low-Resource Nepali Speech Recognition

**Authors:** Aadarsh Pandit, Yudhin Khanal, Ishan Pandey, Kushal Kunwar and [Sunil Regmi](https://md.research.elamur.ai/free/en/authors/sunil-regmi/) · **Year:** 2026 · **Venue:** [Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)](https://md.research.elamur.ai/free/en/venues/chipsal/) ([workshop on Challenges in Processing South Asian Languages 2026](https://md.research.elamur.ai/free/en/venues/chipsal/2026/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.chipsal-1.16/) · [PDF](https://aclanthology.org/2026.chipsal-1.16.pdf) · [DOI 10.63317/45u9hjmchyhh](https://doi.org/10.63317/45u9hjmchyhh) · [OpenAlex](https://openalex.org/W7166566225)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/pandit-2026-reward-guided-fine-tuning-whisper/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses a practical obstacle in Nepali ASR: most usable training audio comes from YouTube with auto-generated SRT subtitles of uneven accuracy, and naive fine-tuning absorbs the errors along with the correct material. The authors place a lightweight, human-informed filter in front of training. They first fine-tune Whisper Small on a pool of 40,000 clips. Its transcriptions of OpenSLR54 audio are then rated by team members in a custom web tool, yielding 2,000 labeled reference–hypothesis pairs. A Random Forest learns to predict these ratings from four cheap metrics. In each later round, the current model re-transcribes the pool, the classifier drops clips predicted as Bad, and Whisper is retrained on what remains. After two rounds, error rates fall noticeably below the plain fine-tuning baseline. The approach matters because it brings human quality judgments into training without full RLHF, and because the authors release a sizeable Nepali speech corpus.

## Key points
- **Model and annotation:** Whisper Small (244M parameters) was the base model. Three labels were used: Good, Neutral and Bad. On a 10% double-labeled subset, annotators agreed 83% of the time, and most disagreements were between Good and Neutral rather than involving Bad.
- **Reward model:** A scikit-learn Random Forest with default settings was trained on WER, CER, length_ratio and length_diff. It reached 81% held-out accuracy, with the two error rates accounting for roughly 65% of feature importance. An MLP alternative overfit.
- **Results:** Two filter-and-retrain cycles lowered WER to 4.89% (down from 5.60%) and brought CER down to 4.52% (previously 5.10%), an 11–13% relative improvement over the authors' own baseline.
- **Limitation:** During filtering, the features are computed against the noisy SRT subtitles. As a result, correct model outputs paired with wrong subtitles may be discarded.
- **Released resources:** The corpus contains 49,216 clips (about 68.4 h, 18.9 GB) drawn from news, podcasts and serials. It is available on Hugging Face (`Aadarsh17/nepali-asr-dataset-public`, CC BY-NC 4.0), and the code is on GitHub (`Aadarshttech/nepali-asr-reward-guided`).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Kathmandu University](https://md.research.elamur.ai/free/en/universities/kathmandu-university/) — University, Kathmandu, Nepal · 7 works · 20 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 493 works

🧭 Topics (2): [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 519 works · [🌱 Low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/) · 287 works

## Cite

Cite as: Aadarsh Pandit, Yudhin Khanal, Ishan Pandey, Kushal Kunwar and Sunil Regmi. 2026. Reward-Guided Fine-Tuning of Whisper for Low-Resource Nepali Speech Recognition. Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{pandit2026reward,
  title = {{Reward-Guided Fine-Tuning of Whisper for Low-Resource Nepali Speech Recognition}},
  author = {Aadarsh Pandit, Yudhin Khanal, Ishan Pandey, Kushal Kunwar and Sunil Regmi},
  booktitle = {Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)},
  year = {2026},
  doi = {10.63317/45u9hjmchyhh},
  url = {https://doi.org/10.63317/45u9hjmchyhh}
}
```

- [← previous Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages](https://md.research.elamur.ai/free/en/papers/yari-2026-revisiting-metric-reliability-fine-grained/)
- [next → Rhetorical Questions in LLM Representations: A Linear Probing Study](https://md.research.elamur.ai/free/en/papers/yao-2026-rhetorical-questions-llm-representations-linear/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/16/)

## References

17 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Baevski, H. Zhou, A. Mohamed, and M. Auli. 2020. [wav2vec 2.0: A framework for self-supervised learning of speech representations](https://md.research.elamur.ai/free/en/papers/baevski-2020-wav2vec-2-0-framework-self/). In Advances in Neural Information Processing Systems, volume 33, pages 12449–12460. Curran Associates Inc.
2. J. Banjara, K. R. Mishra, J. Rathi, K. Karki, and S. Shakya. 2020. Nepali speech recognition using CNN and sequence models. In 2020 IEEE International Conference on Machine Learning and Applied Network Technologies (ICMLANT), pages 1–5. IEEE. [find](https://scholar.google.com/scholar?q=J%2E+Banjara%2C+K%2E+R%2E+Mishra%2C+J%2E+Rathi%2C+K%2E+Karki%2C+and+S%2E+Shakya%2E+2020%2E+Nepali+speech+recognition+using+CNN+and+sequence+models%2E+In+2020+IEEE+International+Conference+on+Machine+Learning+and+Applied+Networ)
3. M. Dhakal, A. Chhetri, A. K. Gupta, P. Lamichhane, S. Pandey, and S. Shakya. 2022. Automatic speech recognition for the Nepali language using CNN, bidirectional LSTM and ResNet. In 2022 International Conference on Inventive Computation Technologies (ICICT), pages 515– 521. IEEE. [find](https://scholar.google.com/scholar?q=M%2E+Dhakal%2C+A%2E+Chhetri%2C+A%2E+K%2E+Gupta%2C+P%2E+Lamichhane%2C+S%2E+Pandey%2C+and+S%2E+Shakya%2E+2022%2E+Automatic+speech+recognition+for+the+Nepali+language+using+CNN%2C+bidirectional+LSTM+and+ResNet%2E+In+2022+International+)
4. R. R. Ghimire, B. K. Bal, and P. Poudyal. 2023. Active learning approach for fine-tuning pre-trained ASR model for a low-resourced language. In Proceedings of the 20th International Conference on Natural Language Processing (ICON 2023), pages 82–89. NLP Association of India. [find](https://scholar.google.com/scholar?q=R%2E+R%2E+Ghimire%2C+B%2E+K%2E+Bal%2C+and+P%2E+Poudyal%2E+2023%2E+Active+learning+approach+for+fine%2Dtuning+pre%2Dtrained+ASR+model+for+a+low%2Dresourced+language%2E+In+Proceedings+of+the+20th+International+Conference+on+Natu)
5. W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed. 2021. Hu-BERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460. [find](https://scholar.google.com/scholar?q=W%2E%2DN%2E+Hsu%2C+B%2E+Bolte%2C+Y%2E%2DH%2E+H%2E+Tsai%2C+K%2E+Lakhotia%2C+R%2E+Salakhutdinov%2C+and+A%2E+Mohamed%2E+2021%2E+Hu%2DBERT%3A+Self%2Dsupervised+speech+representation+learning+by+masked+prediction+of+hidden+units%2E+IEEE%2FACM+Transact)
6. O. Kjartansson, S. Sarin, K. Pipatsrisawat, M. Jansche, and L. Ha. 2018. Crowd-sourced speech corpora for Javanese, Sundanese, Sinhala, Nepali, and Bangladeshi Bengali. In Proceedings of the 6th Workshop on Spoken Language Technologies for Under-Resourced Languages (SLTU 2018), pages 52–55. ISCA. [find](https://scholar.google.com/scholar?q=O%2E+Kjartansson%2C+S%2E+Sarin%2C+K%2E+Pipatsrisawat%2C+M%2E+Jansche%2C+and+L%2E+Ha%2E+2018%2E+Crowd%2Dsourced+speech+corpora+for+Javanese%2C+Sundanese%2C+Sinhala%2C+Nepali%2C+and+Bangladeshi+Bengali%2E+In+Proceedings+of+the+6th+Works)
7. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/).…
8. V. Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. Fazel-Zarandi, A. Baevski, Y. Adi, X. Zhang, W.-N. Hsu, A. Conneau, and M. Auli. 2023. [Scaling speech technology to 1,000+ languages](https://md.research.elamur.ai/free/en/papers/pratap-2023-scaling-speech-technology-1-000/). Journal of Machine Learning Research, 25(97):1–52.
9. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. 2023. [Robust speech recognition via large-scale weak supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/). In Proceedings of the 40th International Conference on Machine Learning (ICML 2023), volume 202, pages 28492–28518. PMLR.
10. P. Regmi, A. Dahal, and B. Joshi. 2019. Nepali speech recognition using RNN-CTC model. International Journal of Computer Applications, 178(31):1–6. [find](https://scholar.google.com/scholar?q=P%2E+Regmi%2C+A%2E+Dahal%2C+and+B%2E+Joshi%2E+2019%2E+Nepali+speech+recognition+using+RNN%2DCTC+model%2E+International+Journal+of+Computer+Applications%2C+178%2831%29%3A1%E2%80%936%2E)
11. S. Regmi and B. K. Bal. 2021. An end-to-end speech recognition for the Nepali language. In Proceedings of the 18th International Conference on Natural Language Processing (ICON 2021), pages 180–185. NLP Association of India. [find](https://scholar.google.com/scholar?q=S%2E+Regmi+and+B%2E+K%2E+Bal%2E+2021%2E+An+end%2Dto%2Dend+speech+recognition+for+the+Nepali+language%2E+In+Proceedings+of+the+18th+International+Conference+on+Natural+Language+Processing+%28ICON+2021%29%2C+pages+180%E2%80%93185%2E+N)
12. A. Conneau, R. Collobert, G. Synnaeve, and M. Auli. 2021. Self-training and pre-training are complementary for speech recognition. In ICASSP 2021 – IEEE International Conference on Acoustics, Speech and Signal Processing, pages 3030–3034. IEEE. [find](https://scholar.google.com/scholar?q=A%2E+Conneau%2C+R%2E+Collobert%2C+G%2E+Synnaeve%2C+and+M%2E+Auli%2E+2021%2E+Self%2Dtraining+and+pre%2Dtraining+are+complementary+for+speech+recognition%2E+In+ICASSP+2021+%E2%80%93+IEEE+International+Conference+on+Acoustics%2C+Speech+a)
13. S. Rijal, S. Adhikari, M. Dahal, M. Awale, and V. Ojha. 2024. Whisper fine-tuning on Nepali language. arXiv preprint arXiv:2411.12587. — [arXiv:2411.12587](https://arxiv.org/abs/2411.12587) [find](https://scholar.google.com/scholar?q=S%2E+Rijal%2C+S%2E+Adhikari%2C+M%2E+Dahal%2C+M%2E+Awale%2C+and+V%2E+Ojha%2E+2024%2E+Whisper+fine%2Dtuning+on+Nepali+language%2E+arXiv+preprint+arXiv%3A2411%2E12587%2E+%E2%80%94+%5BarXiv%3A2411%2E12587%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2411%2E12587%29)
14. Z. Zheng, Z. Ma, Y. Wang, and X. Chen. 2023. Unsupervised active learning: Optimizing labeling cost-effectiveness for automatic speech recognition. arXiv preprint arXiv:2308.14814. — [arXiv:2308.14814](https://arxiv.org/abs/2308.14814) [find](https://scholar.google.com/scholar?q=Z%2E+Zheng%2C+Z%2E+Ma%2C+Y%2E+Wang%2C+and+X%2E+Chen%2E+2023%2E+Unsupervised+active+learning%3A+Optimizing+labeling+cost%2Deffectiveness+for+automatic+speech+recognition%2E+arXiv+preprint+arXiv%3A2308%2E14814%2E+%E2%80%94+%5BarXiv%3A2308%2E14814)
15. S. Singh, F. Hou, and R. Wang. 2023. A novel self-training approach for low-resource speech recognition. In Proceedings of INTERSPEECH 2023, pages 1588–1592. ISCA. [find](https://scholar.google.com/scholar?q=S%2E+Singh%2C+F%2E+Hou%2C+and+R%2E+Wang%2E+2023%2E+A+novel+self%2Dtraining+approach+for+low%2Dresource+speech+recognition%2E+In+Proceedings+of+INTERSPEECH+2023%2C+pages+1588%E2%80%931592%2E+ISCA%2E)
16. Q. Xu, T. Likhomanenko, J. Kahn, A. Hannun, G. Synnaeve, and R. Collobert. 2020. Iterative pseudo-labeling for speech recognition. In Proceedings of INTERSPEECH 2020, pages 1006– 1010. ISCA. [find](https://scholar.google.com/scholar?q=Q%2E+Xu%2C+T%2E+Likhomanenko%2C+J%2E+Kahn%2C+A%2E+Hannun%2C+G%2E+Synnaeve%2C+and+R%2E+Collobert%2E+2020%2E+Iterative+pseudo%2Dlabeling+for+speech+recognition%2E+In+Proceedings+of+INTERSPEECH+2020%2C+pages+1006%E2%80%93+1010%2E+ISCA%2E)
17. Q. Xu, A. Baevski, T. Likhomanenko, P. Tomasello, Language Resource References Oddur Kjartansson, Supheakmungkol Sarin, Knot Pipatsrisawat, Martin Jansche, and Linne Ha. 2018. OpenSLR Nepali Speech Dataset (SLR54). In Proceedings of the 6th Workshop on Spoken Language Technologies for Under-Resource… [find](https://scholar.google.com/scholar?q=Q%2E+Xu%2C+A%2E+Baevski%2C+T%2E+Likhomanenko%2C+P%2E+Tomasello%2C+Language+Resource+References+Oddur+Kjartansson%2C+Supheakmungkol+Sarin%2C+Knot+Pipatsrisawat%2C+Martin+Jansche%2C+and+Linne+Ha%2E+2018%2E+OpenSLR+Nepali+Speech+Da)

*Updated 31 October 2026*