---
title: "AequeVox — paper by Udeshi; and Design) (2021)"
description: "It defines fairness as equal quality of service when conditions change. 60 references, cited by 2 papers here."
authors: ["Sai Sathiesh Rajan; Sakshi Udeshi;", "Sudipta Chattopadhyay   Singapore University of Technology; Design)"]
year: 2021
venue: "arXiv:2110.09843"
source: "https://arxiv.org/abs/2110.09843"
doi: "10.48550/arxiv.2110.09843"
arxiv: "2110.09843"
licence: "CC BY-NC-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rajan-2021-aequevox-automated-fairness-testing-speech/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rajan-2021-aequevox-automated-fairness-testing-speech/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/6/) / AequeVox: Automated Fairness Testing of Speech Recognition Systems

# AequeVox: Automated Fairness Testing of Speech Recognition Systems

**Authors:** Sai Sathiesh Rajan; Sakshi Udeshi; and Sudipta Chattopadhyay   Singapore University of Technology; Design) · **Year:** 2021 · **Venue:** [arXiv:2110.09843](https://md.research.elamur.ai/free/en/venues/lecture-notes-in-computer-science/) ([Lecture notes in computer science 2021](https://md.research.elamur.ai/free/en/venues/lecture-notes-in-computer-science/2021/)) · **Publisher:** [Springer Science+Business Media](https://md.research.elamur.ai/free/en/publishers/springer-science-business-media/) · **Licence:** [CC BY-NC-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2110.09843) · [PDF](https://arxiv.org/pdf/2110.09843) · [DOI 10.48550/arxiv.2110.09843](https://doi.org/10.48550/arxiv.2110.09843) · [arXiv 2110.09843](https://arxiv.org/abs/2110.09843) · [OpenAlex](https://openalex.org/W3206378830)

**Citations:** 11 citations (OpenAlex, 2026-10-14) · 2 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Speech recognition services reach very diverse users, but checking whether they work equally well for every group usually requires hand-made transcripts. AequeVox is a black-box testing framework that measures ASR fairness without ground-truth text. It defines fairness as equal quality of service when conditions change. Recordings from two speaker groups are passed through eight signal-level metamorphic transformations, such as added noise, dropped segments and low- or high-pass filtering. The framework then compares how much the error rate rises for each group. If one group degrades more than the other by more than a threshold τ, it reports a fairness violation. A companion fault-localisation step flags words that tend to break under these transformations. The authors tested Google Cloud, Microsoft Azure and IBM Watson on four corpora of human speech. Non-native, female and Nigerian English speakers suffered 109%, 528.5% and 156.9% more errors than native, male and UK Midlands speakers, respectively. A user study confirmed that the transformed audio stays understandable to people. This matters because ASR increasingly powers assistive and safety-critical tools.

## Key points
- Fairness criterion: |ASR_Err(GR_i) − ASR_Err(GR_j)| ≤ τ, applied to the *increase* in error after a transformation rather than to raw error. The authors link this to Equality of Outcomes and group fairness.
- Data: Speech Accent Archive, RAVDESS, a Midlands corpus of British Isles accents, and a Nigerian English speech dataset. The systems under test were Google Cloud, Microsoft Azure and IBM Watson.
- User study: 82.9% of transformations scored above 7/10 for comprehensibility, and the lowest score was 6.78.
- Words predicted to be non-robust showed 223.8% more errors than words predicted to be robust, across all ASRs. This was checked on TTS speech built from random grammar-generated sentences.
- The authors deliberately avoid TTS as the test input, because human speech carries accent, gender, age and intonation cues that synthesis cannot reproduce. Code and data are public on GitHub (sparkssss/AequeVox).

## Interesting ideas
- The method judges fairness by how unequally groups degrade under the same controlled perturbation, not by absolute error, which removes the need for gold labels.
- Testing is differential: the system's behaviour on the unperturbed input is the comparison point, so any unlabelled corpus becomes test material.
- The authors run a comprehensibility study on the transformations themselves, so that reported failures come from inputs humans can still understand.
- Fault localisation is validated separately: flagged fragile words are placed into random grammar-generated sentences, and the test checks that they still fail more often there.
- The authors note that degradation-based measures only work when baseline accuracy is high, since a weak system has little room left to degrade.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Apply identical perturbations (typos, casing loss, dialect spellings) to sentences from different genres or registers and flag any genre whose parse quality drops disproportionately.
- **I2** (tree): Parse each sentence and a meaning-preserving variant with gparse and the perceptron parser, then treat tree disagreement as a gold-free robustness signal.
- **I3** (learn): Before trusting perturbed test sentences, have an LLM teacher tier or a human sample confirm meaning is preserved, as a control against invalid stress tests.
- **I4** (tokens): Insert words the tagger or lemmatiser marks as unstable into grammar-generated carrier sentences to confirm the weakness is lexical and not contextual.
- **I5** (answer): Run perturbation-based robustness checks on reader tests only where baseline accuracy is already high, and report the baseline next to each robustness figure.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Singapore University of Technology and Design](https://md.research.elamur.ai/free/en/universities/singapore-university-of-technology-and-design/) — University, Singapore, Singapore · 49 works · 11,074 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 543 works · [💻 Software](https://md.research.elamur.ai/free/en/shelves/computer-science/software/) · 92 works

🧭 Topics (2): [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 517 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 480 works

## Cite

Cite as: Sai Sathiesh Rajan; Sakshi Udeshi; and Sudipta Chattopadhyay   Singapore University of Technology; Design). 2021. AequeVox: Automated Fairness Testing of Speech Recognition Systems. arXiv:2110.09843.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{udeshi2021aequevox,
  title = {{AequeVox: Automated Fairness Testing of Speech Recognition Systems}},
  author = {Sai Sathiesh Rajan; Sakshi Udeshi; and Sudipta Chattopadhyay Singapore University of Technology; Design)},
  year = {2021},
  doi = {10.48550/arxiv.2110.09843},
  eprint = {2110.09843},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2110.09843}
}
```

- [← previous "I'll be there for you": The One with Understanding Indirect Answers](https://md.research.elamur.ai/free/en/papers/damgaard-2021-i-ll-be-there-you/)
- [next → Machine Learning-Based Approach for Arabic Dialect Identification](https://md.research.elamur.ai/free/en/papers/nayel-2021-machine-learning-based-approach-arabic/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/6/)

## References

60 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Audio data augmentation (2021), [https://www.kaggle.com/CVxTz/audio-data-augmentation](https://www.kaggle.com/CVxTz/audio-data-augmentation) [↗](https://www.kaggle.com/CVxTz/audio-data-augmentation)
2. Crowdsourced high-quality nigerian english speech data set (2021), [http://openslr.org/70/](http://openslr.org/70/) [↗](http://openslr.org/70/)
3. Grammarly (2021), [https://app.grammarly.com/](https://app.grammarly.com/) [↗](https://app.grammarly.com/)
4. Aggarwal, A., Lohia, P., Nagar, S., Dey, K., Saha, D.: Black box fairness testing of machine learning models. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. pp. 625–635 (2019) [find](https://scholar.google.com/scholar?q=Aggarwal%2C+A%2E%2C+Lohia%2C+P%2E%2C+Nagar%2C+S%2E%2C+Dey%2C+K%2E%2C+Saha%2C+D%2E%3A+Black+box+fairness+testing+of+machine+learning+models%2E+In%3A+Proceedings+of+the+2019+27th+ACM+Joint+Meeting+on+European+Software+Engineering+Confer)
5. Asyrofi, M.H., Thung, F., Lo, D., Jiang, L.: Crossasr: Efficient differential testing of automatic speech recognition via text-to-speech. In: 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). pp. 640–650 (2020). https://doi.org/10.1109/ICSME46990.2020.00066 [↗](https://doi.org/10.1109/icsme46990.2020.00066)
6. Buolamwini, J., Gebru, T.: Gender shades: Intersectional accuracy disparities in commercial gender classification. In: Conference on fairness, accountability and transparency. pp. 77–91. PMLR (2018) [find](https://scholar.google.com/scholar?q=Buolamwini%2C+J%2E%2C+Gebru%2C+T%2E%3A+Gender+shades%3A+Intersectional+accuracy+disparities+in+commercial+gender+classification%2E+In%3A+Conference+on+fairness%2C+accountability+and+transparency%2E+pp%2E+77%E2%80%9391%2E+PMLR+%282018%29)
7. Butterworth, S., et al.: On the theory of filter amplifiers. Wireless Engineer 7(6), 536–541 (1930) [find](https://scholar.google.com/scholar?q=Butterworth%2C+S%2E%2C+et+al%2E%3A+On+the+theory+of+filter+amplifiers%2E+Wireless+Engineer+7%286%29%2C+536%E2%80%93541+%281930%29)
8. Calò, A., Arcaini, P., Ali, S., Hauer, F., Ishikawa, F.: Simultaneously searching and solving multiple avoidable collisions for testing autonomous driving systems. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference. pp. 1055–1063 (2020) [find](https://scholar.google.com/scholar?q=Cal%C3%B2%2C+A%2E%2C+Arcaini%2C+P%2E%2C+Ali%2C+S%2E%2C+Hauer%2C+F%2E%2C+Ishikawa%2C+F%2E%3A+Simultaneously+searching+and+solving+multiple+avoidable+collisions+for+testing+autonomous+driving+systems%2E+In%3A+Proceedings+of+the+2020+Genetic+)
9. Carlini, N., Wagner, D.: Audio adversarial examples: Targeted attacks on speech-to-text. In: 2018 IEEE Security and Privacy Workshops (SPW). pp. 1–7. IEEE (2018) [find](https://scholar.google.com/scholar?q=Carlini%2C+N%2E%2C+Wagner%2C+D%2E%3A+Audio+adversarial+examples%3A+Targeted+attacks+on+speech%2Dto%2Dtext%2E+In%3A+2018+IEEE+Security+and+Privacy+Workshops+%28SPW%29%2E+pp%2E+1%E2%80%937%2E+IEEE+%282018%29)
10. Chen, G., Chen, S., Fan, L., Du, X., Zhao, Z., Song, F., Liu, Y.: Who is real bob? adversarial attacks on speaker recognition systems. In: IEEE Symposium on Security and Privacy (2021) [find](https://scholar.google.com/scholar?q=Chen%2C+G%2E%2C+Chen%2C+S%2E%2C+Fan%2C+L%2E%2C+Du%2C+X%2E%2C+Zhao%2C+Z%2E%2C+Song%2C+F%2E%2C+Liu%2C+Y%2E%3A+Who+is+real+bob%3F+adversarial+attacks+on+speaker+recognition+systems%2E+In%3A+IEEE+Symposium+on+Security+and+Privacy+%282021%29)
11. Demirsahin, I., Kjartansson, O., Gutkin, A., Rivera, C.: [Open-source Multi-speaker Corpora of the English Accents in the British Isles](https://md.research.elamur.ai/free/en/papers/demirsahin-2020-open-source-multi-speaker-corpora/). In: Proceedings of The 12th Language Resources and Evaluation Conference (LREC). pp. 6532–6541. European Language Resources Association (ELRA), Marseille, France (May 2020), https://www.aclweb.org/anthology/2020.lrec-1.804 [↗](https://aclanthology.org/2020.lrec-1.804/)
12. Denton, E., Hutchinson, B., Mitchell, M., Gebru, T., Zaldivar, A.: Image counterfactual sensitivity analysis for detecting unintended bias (2019) [find](https://scholar.google.com/scholar?q=Denton%2C+E%2E%2C+Hutchinson%2C+B%2E%2C+Mitchell%2C+M%2E%2C+Gebru%2C+T%2E%2C+Zaldivar%2C+A%2E%3A+Image+counterfactual+sensitivity+analysis+for+detecting+unintended+bias+%282019%29)
13. Du, X., Xie, X., Li, Y., Ma, L., Zhao, J., Liu, Y.: Deepcruiser: Automated guided testing for stateful deep learning systems (2018) [find](https://scholar.google.com/scholar?q=Du%2C+X%2E%2C+Xie%2C+X%2E%2C+Li%2C+Y%2E%2C+Ma%2C+L%2E%2C+Zhao%2C+J%2E%2C+Liu%2C+Y%2E%3A+Deepcruiser%3A+Automated+guided+testing+for+stateful+deep+learning+systems+%282018%29)
14. Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fairness through awareness. In: Proceedings of the 3rd innovations in theoretical computer science conference. pp. 214–226 (2012) [find](https://scholar.google.com/scholar?q=Dwork%2C+C%2E%2C+Hardt%2C+M%2E%2C+Pitassi%2C+T%2E%2C+Reingold%2C+O%2E%2C+Zemel%2C+R%2E%3A+Fairness+through+awareness%2E+In%3A+Proceedings+of+the+3rd+innovations+in+theoretical+computer+science+conference%2E+pp%2E+214%E2%80%93226+%282012%29)
15. Eniser, H.F., Gerasimou, S., Sen, A.: Deepfault: Fault localization for deep neural networks. In: Hähnle, R., van der Aalst, W.M.P. (eds.) Fundamental Approaches to Software Engineering - 22nd International Conference, FASE 2019, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2019, Prague, Czech Republic, April 6-11, 2019, Proceedings. Lecture Notes in Computer Science, vol. 11424, pp. 171–191. Springer (2019) AequeVox S. Rajan et al. [find](https://scholar.google.com/scholar?q=Eniser%2C+H%2EF%2E%2C+Gerasimou%2C+S%2E%2C+Sen%2C+A%2E%3A+Deepfault%3A+Fault+localization+for+deep+neural+networks%2E+In%3A+H%C3%A4hnle%2C+R%2E%2C+van+der+Aalst%2C+W%2EM%2EP%2E+%28eds%2E%29+Fundamental+Approaches+to+Software+Engineering+%2D+22nd+Interna)
16. Feng, Y., Shi, Q., Gao, X., Wan, J., Fang, C., Chen, Z.: Deepgini: prioritizing massive tests to enhance the robustness of deep neural networks. In: Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. pp. 177–188 (2020) [find](https://scholar.google.com/scholar?q=Feng%2C+Y%2E%2C+Shi%2C+Q%2E%2C+Gao%2C+X%2E%2C+Wan%2C+J%2E%2C+Fang%2C+C%2E%2C+Chen%2C+Z%2E%3A+Deepgini%3A+prioritizing+massive+tests+to+enhance+the+robustness+of+deep+neural+networks%2E+In%3A+Proceedings+of+the+29th+ACM+SIGSOFT+International+S)
17. Galhotra, S., Brun, Y., Meliou, A.: Fairness testing: testing software for discrimination. In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, September 4-8, 2017. pp. 498–510 (2017). https://doi.org/10.1145/3106237.3106277, http://doi.acm. org/10.1145/3106237.3106277 [↗](https://doi.org/10.1145/3106237.3106277)
18. Goss, F.R., Zhou, L., Weiner, S.G.: Incidence of speech recognition errors in the emergency department. International journal of medical informatics 93, 70–73 (2016) [find](https://scholar.google.com/scholar?q=Goss%2C+F%2ER%2E%2C+Zhou%2C+L%2E%2C+Weiner%2C+S%2EG%2E%3A+Incidence+of+speech+recognition+errors+in+the+emergency+department%2E+International+journal+of+medical+informatics+93%2C+70%E2%80%9373+%282016%29)
19. Guo, Q., Xie, X., Li, Y., Zhang, X., Liu, Y., Li, X., Shen, C.: Audee: Automated testing for deep learning frameworks. In: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering (ASE). pp. 486–498. ACM (Dec 2020) [find](https://scholar.google.com/scholar?q=Guo%2C+Q%2E%2C+Xie%2C+X%2E%2C+Li%2C+Y%2E%2C+Zhang%2C+X%2E%2C+Liu%2C+Y%2E%2C+Li%2C+X%2E%2C+Shen%2C+C%2E%3A+Audee%3A+Automated+testing+for+deep+learning+frameworks%2E+In%3A+Proceedings+of+the+35th+IEEE%2FACM+International+Conference+on+Automated+Softwa)
20. Hawley, M.S.: Speech recognition as an input to electronic assistive technology. British Journal of Occupational Therapy 65(1), 15–20 (2002) [find](https://scholar.google.com/scholar?q=Hawley%2C+M%2ES%2E%3A+Speech+recognition+as+an+input+to+electronic+assistive+technology%2E+British+Journal+of+Occupational+Therapy+65%281%29%2C+15%E2%80%9320+%282002%29)
21. Helmke, H., Ohneiser, O., Mühlhausen, T., Wies, M.: Reducing controller workload with automatic speech recognition. In: 2016 IEEE/AIAA 35th Digital Avionics Systems Conference (DASC). pp. 1–10. IEEE (2016) [find](https://scholar.google.com/scholar?q=Helmke%2C+H%2E%2C+Ohneiser%2C+O%2E%2C+M%C3%BChlhausen%2C+T%2E%2C+Wies%2C+M%2E%3A+Reducing+controller+workload+with+automatic+speech+recognition%2E+In%3A+2016+IEEE%2FAIAA+35th+Digital+Avionics+Systems+Conference+%28DASC%29%2E+pp%2E+1%E2%80%9310%2E+IEEE+%28)
22. Huang, C., Chen, T., Li, S.Z., Chang, E., Zhou, J.L.: Analysis of speaker variability. In: INTERSPEECH. pp. 1377–1380 (2001) [find](https://scholar.google.com/scholar?q=Huang%2C+C%2E%2C+Chen%2C+T%2E%2C+Li%2C+S%2EZ%2E%2C+Chang%2C+E%2E%2C+Zhou%2C+J%2EL%2E%3A+Analysis+of+speaker+variability%2E+In%3A+INTERSPEECH%2E+pp%2E+1377%E2%80%931380+%282001%29)
23. Huber, D.M., Runstein, R.E.: Modern recording techniques, pp. 416,487. CRC Press (2013) [find](https://scholar.google.com/scholar?q=Huber%2C+D%2EM%2E%2C+Runstein%2C+R%2EE%2E%3A+Modern+recording+techniques%2C+pp%2E+416%2C487%2E+CRC+Press+%282013%29)
24. Iwama, F., Fukuda, T.: Automated testing of basic recognition capability for speech recognition systems. In: 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST). pp. 13–24. IEEE (2019) [find](https://scholar.google.com/scholar?q=Iwama%2C+F%2E%2C+Fukuda%2C+T%2E%3A+Automated+testing+of+basic+recognition+capability+for+speech+recognition+systems%2E+In%3A+2019+12th+IEEE+Conference+on+Software+Testing%2C+Validation+and+Verification+%28ICST%29%2E+pp%2E+13%E2%80%932)
25. Jain, A., Upreti, M., Jyothi, P.: Improved accented speech recognition using accent embeddings and multi-task learning. In: Interspeech. pp. 2454–2458 (2018) [find](https://scholar.google.com/scholar?q=Jain%2C+A%2E%2C+Upreti%2C+M%2E%2C+Jyothi%2C+P%2E%3A+Improved+accented+speech+recognition+using+accent+embeddings+and+multi%2Dtask+learning%2E+In%3A+Interspeech%2E+pp%2E+2454%E2%80%932458+%282018%29)
26. Johnson, D.H.: Signal-to-noise ratio. Scholarpedia 1(12), 2088 (2006) [find](https://scholar.google.com/scholar?q=Johnson%2C+D%2EH%2E%3A+Signal%2Dto%2Dnoise+ratio%2E+Scholarpedia+1%2812%29%2C+2088+%282006%29)
27. Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C., Rickford, J.R., Jurafsky, D., Goel, S.: Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences 117(14), 7684–7689 (2020) [find](https://scholar.google.com/scholar?q=Koenecke%2C+A%2E%2C+Nam%2C+A%2E%2C+Lake%2C+E%2E%2C+Nudell%2C+J%2E%2C+Quartey%2C+M%2E%2C+Mengesha%2C+Z%2E%2C+Toups%2C+C%2E%2C+Rickford%2C+J%2ER%2E%2C+Jurafsky%2C+D%2E%2C+Goel%2C+S%2E%3A+Racial+disparities+in+automated+speech+recognition%2E+Proceedings+of+the+Nation)
28. Kopald, H.D., Chanen, A., Chen, S., Smith, E.C., Tarakan, R.M.: Applying automatic speech recognition technology to air traffic management. In: 2013 IEEE/AIAA 32nd Digital Avionics Systems Conference (DASC). pp. 6C3–1. IEEE (2013) [find](https://scholar.google.com/scholar?q=Kopald%2C+H%2ED%2E%2C+Chanen%2C+A%2E%2C+Chen%2C+S%2E%2C+Smith%2C+E%2EC%2E%2C+Tarakan%2C+R%2EM%2E%3A+Applying+automatic+speech+recognition+technology+to+air+traffic+management%2E+In%3A+2013+IEEE%2FAIAA+32nd+Digital+Avionics+Systems+Conference+)
29. Kreuk, F., Adi, Y., Cisse, M., Keshet, J.: Fooling end-to-end speaker verification with adversarial examples. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1962–1966. IEEE (2018) [find](https://scholar.google.com/scholar?q=Kreuk%2C+F%2E%2C+Adi%2C+Y%2E%2C+Cisse%2C+M%2E%2C+Keshet%2C+J%2E%3A+Fooling+end%2Dto%2Dend+speaker+verification+with+adversarial+examples%2E+In%3A+2018+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29)
30. Levenshtein, V.I.: Binary codes capable of correcting deletions, insertions, and reversals. In: Soviet physics doklady. vol. 10, pp. 707–710. Soviet Union (1966) [find](https://scholar.google.com/scholar?q=Levenshtein%2C+V%2EI%2E%3A+Binary+codes+capable+of+correcting+deletions%2C+insertions%2C+and+reversals%2E+In%3A+Soviet+physics+doklady%2E+vol%2E+10%2C+pp%2E+707%E2%80%93710%2E+Soviet+Union+%281966%29)
31. Li, J., Deng, L., Gong, Y., Haeb-Umbach, R.: An overview of noise-robust automatic speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing 22(4), 745–777 (2014) [find](https://scholar.google.com/scholar?q=Li%2C+J%2E%2C+Deng%2C+L%2E%2C+Gong%2C+Y%2E%2C+Haeb%2DUmbach%2C+R%2E%3A+An+overview+of+noise%2Drobust+automatic+speech+recognition%2E+IEEE%2FACM+Transactions+on+Audio%2C+Speech%2C+and+Language+Processing+22%284%29%2C+745%E2%80%93777+%282014%29)
32. Livingstone, S.R., Russo, F.A.: The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one 13(5), e0196391 (2018) [find](https://scholar.google.com/scholar?q=Livingstone%2C+S%2ER%2E%2C+Russo%2C+F%2EA%2E%3A+The+ryerson+audio%2Dvisual+database+of+emotional+speech+and+song+%28ravdess%29%3A+A+dynamic%2C+multimodal+set+of+facial+and+vocal+expressions+in+north+american+english%2E+PloS+one+)
33. Lundberg, S.M., Lee, S.I.: [A unified approach to interpreting model predictions](https://md.research.elamur.ai/free/en/papers/lundberg-2017-unified-approach-interpreting-model-predictions/). In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems 30, pp. 4765–4774. Curran Associates, Inc. (2017), [http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf](http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf) [↗](http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf)
34. Ma, L., Juefei-Xu, F., Zhang, F., Sun, J., Xue, M., Li, B., Chen, C., Su, T., Li, L., Liu, Y., Zhao, J., Wang, Y.: Deepgauge: multi-granularity testing criteria for deep learning systems. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, September 3-7, 2018. pp. 120–131 (2018) [find](https://scholar.google.com/scholar?q=Ma%2C+L%2E%2C+Juefei%2DXu%2C+F%2E%2C+Zhang%2C+F%2E%2C+Sun%2C+J%2E%2C+Xue%2C+M%2E%2C+Li%2C+B%2E%2C+Chen%2C+C%2E%2C+Su%2C+T%2E%2C+Li%2C+L%2E%2C+Liu%2C+Y%2E%2C+Zhao%2C+J%2E%2C+Wang%2C+Y%2E%3A+Deepgauge%3A+multi%2Dgranularity+testing+criteria+for+deep+learning+systems%2E+In%3A+Proceedi)
35. Ma, P., Wang, S., Liu, J.: Metamorphic testing and certified mitigation of fairness violations in NLP models. In: Bessiere, C. (ed.) Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020. pp. 458–465 [find](https://scholar.google.com/scholar?q=Ma%2C+P%2E%2C+Wang%2C+S%2E%2C+Liu%2C+J%2E%3A+Metamorphic+testing+and+certified+mitigation+of+fairness+violations+in+NLP+models%2E+In%3A+Bessiere%2C+C%2E+%28ed%2E%29+Proceedings+of+the+Twenty%2DNinth+International+Joint+Conference+on+A)
36. Odena, A., Olsson, C., Andersen, D., Goodfellow, I.: Tensorfuzz: Debugging neural networks with coverage-guided fuzzing. In: International Conference on Machine Learning. pp. 4901–4911. PMLR (2019) [find](https://scholar.google.com/scholar?q=Odena%2C+A%2E%2C+Olsson%2C+C%2E%2C+Andersen%2C+D%2E%2C+Goodfellow%2C+I%2E%3A+Tensorfuzz%3A+Debugging+neural+networks+with+coverage%2Dguided+fuzzing%2E+In%3A+International+Conference+on+Machine+Learning%2E+pp%2E+4901%E2%80%934911%2E+PMLR+%282019%29)
37. Pei, K., Cao, Y., Yang, J., Jana, S.: Deepxplore: Automated whitebox testing of deep learning systems. In: Proceedings of the 26th Symposium on Operating Systems Principles, Shanghai, China, October 28-31, 2017. pp. 1–18 (2017) [find](https://scholar.google.com/scholar?q=Pei%2C+K%2E%2C+Cao%2C+Y%2E%2C+Yang%2C+J%2E%2C+Jana%2C+S%2E%3A+Deepxplore%3A+Automated+whitebox+testing+of+deep+learning+systems%2E+In%3A+Proceedings+of+the+26th+Symposium+on+Operating+Systems+Principles%2C+Shanghai%2C+China%2C+October+2)
38. Phillips, A.: Defending equality of outcome. Journal of political philosophy 12(1), 1–19 (2004) [find](https://scholar.google.com/scholar?q=Phillips%2C+A%2E%3A+Defending+equality+of+outcome%2E+Journal+of+political+philosophy+12%281%29%2C+1%E2%80%9319+%282004%29)
39. Qin, Y., Carlini, N., Cottrell, G., Goodfellow, I., Raffel, C.: Imperceptible, robust, and targeted adversarial examples for automatic speech recognition. In: International conference on machine learning. pp. 5231–5240. PMLR (2019) [find](https://scholar.google.com/scholar?q=Qin%2C+Y%2E%2C+Carlini%2C+N%2E%2C+Cottrell%2C+G%2E%2C+Goodfellow%2C+I%2E%2C+Raffel%2C+C%2E%3A+Imperceptible%2C+robust%2C+and+targeted+adversarial+examples+for+automatic+speech+recognition%2E+In%3A+International+conference+on+machine+learn)
40. Ribeiro, M.T., Singh, S., Guestrin, C.: ["why should I trust you?": Explaining the predictions of any classifier](https://md.research.elamur.ai/free/en/papers/ribeiro-2016-why-should-i-trust-explaining/). In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016. pp. 1135–1144 (2016)
41. Ribeiro, M.T., Singh, S., Guestrin, C.: Anchors: High-precision model-agnostic explanations. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 32 (2018) [find](https://scholar.google.com/scholar?q=Ribeiro%2C+M%2ET%2E%2C+Singh%2C+S%2E%2C+Guestrin%2C+C%2E%3A+Anchors%3A+High%2Dprecision+model%2Dagnostic+explanations%2E+In%3A+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2E+vol%2E+32+%282018%29)
42. Ribeiro, M.T., Wu, T., Guestrin, C., Singh, S.: Beyond accuracy: Behavioral testing of NLP models with checklist. In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J.R. (eds.) Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020. pp. 4902–4912. Association for Computational Linguistics (2020) [find](https://scholar.google.com/scholar?q=Ribeiro%2C+M%2ET%2E%2C+Wu%2C+T%2E%2C+Guestrin%2C+C%2E%2C+Singh%2C+S%2E%3A+Beyond+accuracy%3A+Behavioral+testing+of+NLP+models+with+checklist%2E+In%3A+Jurafsky%2C+D%2E%2C+Chai%2C+J%2E%2C+Schluter%2C+N%2E%2C+Tetreault%2C+J%2ER%2E+%28eds%2E%29+Proceedings+of+the+58)
43. Sharma, A., Demir, C., Ngomo, A.C.N., Wehrheim, H.: Mlcheck-property-driven testing of machine learning models. arXiv preprint arXiv:2105.00741 (2021) [↗](https://arxiv.org/abs/2105.00741)
44. Sharma, A., Wehrheim, H.: Testing machine learning algorithms for balanced data usage. In: 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST). pp. 125–135. IEEE (2019) [find](https://scholar.google.com/scholar?q=Sharma%2C+A%2E%2C+Wehrheim%2C+H%2E%3A+Testing+machine+learning+algorithms+for+balanced+data+usage%2E+In%3A+2019+12th+IEEE+Conference+on+Software+Testing%2C+Validation+and+Verification+%28ICST%29%2E+pp%2E+125%E2%80%93135%2E+IEEE+%282019%29)
45. Sharma, A., Wehrheim, H.: Automatic fairness testing of machine learning models. In: IFIP International Conference on Testing Software and Systems. pp. 255–271. Springer (2020) [find](https://scholar.google.com/scholar?q=Sharma%2C+A%2E%2C+Wehrheim%2C+H%2E%3A+Automatic+fairness+testing+of+machine+learning+models%2E+In%3A+IFIP+International+Conference+on+Testing+Software+and+Systems%2E+pp%2E+255%E2%80%93271%2E+Springer+%282020%29)
46. Sharma, A., Wehrheim, H.: Higher income, larger loan? monotonicity testing of machine learning models. In: Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. pp. 200–210 (2020) [find](https://scholar.google.com/scholar?q=Sharma%2C+A%2E%2C+Wehrheim%2C+H%2E%3A+Higher+income%2C+larger+loan%3F+monotonicity+testing+of+machine+learning+models%2E+In%3A+Proceedings+of+the+29th+ACM+SIGSOFT+International+Symposium+on+Software+Testing+and+Analysis%2E)
47. Soremekun, E., Udeshi, S., Chattopadhyay, S.: Astraea: Grammar-based fairness testing. arXiv preprint arXiv:2010.02542 (2020) AequeVox S. Rajan et al. [↗](https://arxiv.org/abs/2010.02542)
48. Sun, Y., Chockler, H., Huang, X., Kroening, D.: Explaining image classifiers using statistical fault localization. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J. (eds.) Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII. Lecture Notes in Computer Science, vol. 12373, pp. 391–406. Springer (2020) [find](https://scholar.google.com/scholar?q=Sun%2C+Y%2E%2C+Chockler%2C+H%2E%2C+Huang%2C+X%2E%2C+Kroening%2C+D%2E%3A+Explaining+image+classifiers+using+statistical+fault+localization%2E+In%3A+Vedaldi%2C+A%2E%2C+Bischof%2C+H%2E%2C+Brox%2C+T%2E%2C+Frahm%2C+J%2E+%28eds%2E%29+Computer+Vision+%2D+ECCV+2020+)
49. Sun, Y., Wu, M., Ruan, W., Huang, X., Kwiatkowska, M., Kroening, D.: Concolic testing for deep neural networks. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, September 3-7, 2018. pp. 109–119 (2018) [find](https://scholar.google.com/scholar?q=Sun%2C+Y%2E%2C+Wu%2C+M%2E%2C+Ruan%2C+W%2E%2C+Huang%2C+X%2E%2C+Kwiatkowska%2C+M%2E%2C+Kroening%2C+D%2E%3A+Concolic+testing+for+deep+neural+networks%2E+In%3A+Proceedings+of+the+33rd+ACM%2FIEEE+International+Conference+on+Automated+Software+Engi)
50. Tian, Y., Pei, K., Jana, S., Ray, B.: Deeptest: automated testing of deep-neural-network-driven autonomous cars. In: Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018. pp. 303–314 (2018) [find](https://scholar.google.com/scholar?q=Tian%2C+Y%2E%2C+Pei%2C+K%2E%2C+Jana%2C+S%2E%2C+Ray%2C+B%2E%3A+Deeptest%3A+automated+testing+of+deep%2Dneural%2Dnetwork%2Ddriven+autonomous+cars%2E+In%3A+Proceedings+of+the+40th+International+Conference+on+Software+Engineering%2C+ICSE+2018)
51. Udeshi, S., Arora, P., Chattopadhyay, S.: Automated directed fairness testing. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, September 3-7, 2018. pp. 98–108 (2018) [find](https://scholar.google.com/scholar?q=Udeshi%2C+S%2E%2C+Arora%2C+P%2E%2C+Chattopadhyay%2C+S%2E%3A+Automated+directed+fairness+testing%2E+In%3A+Proceedings+of+the+33rd+ACM%2FIEEE+International+Conference+on+Automated+Software+Engineering%2C+ASE+2018%2C+Montpellier%2C+F)
52. Udeshi, S.S., Chattopadhyay, S.: Grammar based directed testing of machine learning systems. IEEE Transactions on Software Engineering (2019) [find](https://scholar.google.com/scholar?q=Udeshi%2C+S%2ES%2E%2C+Chattopadhyay%2C+S%2E%3A+Grammar+based+directed+testing+of+machine+learning+systems%2E+IEEE+Transactions+on+Software+Engineering+%282019%29)
53. Verma, S., Rubin, J.: Fairness definitions explained. In: 2018 ieee/acm international workshop on software fairness (fairware). pp. 1–7. IEEE (2018) [find](https://scholar.google.com/scholar?q=Verma%2C+S%2E%2C+Rubin%2C+J%2E%3A+Fairness+definitions+explained%2E+In%3A+2018+ieee%2Facm+international+workshop+on+software+fairness+%28fairware%29%2E+pp%2E+1%E2%80%937%2E+IEEE+%282018%29)
54. Wang, J., Chen, J., Sun, Y., Ma, X., Wang, D., Sun, J., Cheng, P.: Robot: Robustness-oriented testing for deep learning systems. In: ICSE ’21: 43rd International Conference on Software Engineering (2021) [find](https://scholar.google.com/scholar?q=Wang%2C+J%2E%2C+Chen%2C+J%2E%2C+Sun%2C+Y%2E%2C+Ma%2C+X%2E%2C+Wang%2C+D%2E%2C+Sun%2C+J%2E%2C+Cheng%2C+P%2E%3A+Robot%3A+Robustness%2Doriented+testing+for+deep+learning+systems%2E+In%3A+ICSE+%E2%80%9921%3A+43rd+International+Conference+on+Software+Engineering+%2820)
55. Wardat, M., Le, W., Rajan, H.: Deeplocalize: Fault localization for deep neural networks. In: 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021. pp. 251–262. IEEE (2021) [find](https://scholar.google.com/scholar?q=Wardat%2C+M%2E%2C+Le%2C+W%2E%2C+Rajan%2C+H%2E%3A+Deeplocalize%3A+Fault+localization+for+deep+neural+networks%2E+In%3A+43rd+IEEE%2FACM+International+Conference+on+Software+Engineering%2C+ICSE+2021%2C+Madrid%2C+Spain%2C+22%2D30+May+2021%2E+)
56. Weik, M.: Communications standard dictionary. Springer Science & Business Media (2012) [find](https://scholar.google.com/scholar?q=Weik%2C+M%2E%3A+Communications+standard+dictionary%2E+Springer+Science+%26+Business+Media+%282012%29)
57. Weinberger, S.H., Kunath, S.A.: The speech accent archive: towards a typology of english accents. In: Corpus-based Studies in Language Use, Language Learning, and Language Documentation, pp. 265–281. Brill Rodopi (2011) [find](https://scholar.google.com/scholar?q=Weinberger%2C+S%2EH%2E%2C+Kunath%2C+S%2EA%2E%3A+The+speech+accent+archive%3A+towards+a+typology+of+english+accents%2E+In%3A+Corpus%2Dbased+Studies+in+Language+Use%2C+Language+Learning%2C+and+Language+Documentation%2C+pp%2E+265%E2%80%93281%2E+)
58. Xie, X., Ma, L., Juefei-Xu, F., Xue, M., Chen, H., Liu, Y., Zhao, J., Li, B., Yin, J., See, S.: Deephunter: a coverage-guided fuzz testing framework for deep neural networks. In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. pp. 146–157 (2019) [find](https://scholar.google.com/scholar?q=Xie%2C+X%2E%2C+Ma%2C+L%2E%2C+Juefei%2DXu%2C+F%2E%2C+Xue%2C+M%2E%2C+Chen%2C+H%2E%2C+Liu%2C+Y%2E%2C+Zhao%2C+J%2E%2C+Li%2C+B%2E%2C+Yin%2C+J%2E%2C+See%2C+S%2E%3A+Deephunter%3A+a+coverage%2Dguided+fuzz+testing+framework+for+deep+neural+networks%2E+In%3A+Proceedings+of+the+28)
59. Xie, X., Zhang, Z., Chen, T.Y., Liu, Y., Poon, P.L., Xu, B.: Mettle: a metamorphic testing approach to assessing and validating unsupervised machine learning systems. IEEE Transactions on Reliability 69(4), 1293–1322 (2020) [find](https://scholar.google.com/scholar?q=Xie%2C+X%2E%2C+Zhang%2C+Z%2E%2C+Chen%2C+T%2EY%2E%2C+Liu%2C+Y%2E%2C+Poon%2C+P%2EL%2E%2C+Xu%2C+B%2E%3A+Mettle%3A+a+metamorphic+testing+approach+to+assessing+and+validating+unsupervised+machine+learning+systems%2E+IEEE+Transactions+on+Reliability+)
60. Zhang, J., Harman, M.: "ignorance and prejudice" in software fairness. In: International Conference on Software Engineering. vol. 43. IEEE (2021) [find](https://scholar.google.com/scholar?q=Zhang%2C+J%2E%2C+Harman%2C+M%2E%3A+%22ignorance+and+prejudice%22+in+software+fairness%2E+In%3A+International+Conference+on+Software+Engineering%2E+vol%2E+43%2E+IEEE+%282021%29)

*Updated 30 October 2026*