---
title: "The Perceptimatic English Benchmark for Speech Perception"
description: "Speech recognizers process natural recorded speech, so they could serve as a starting point for… 27 references, cited by 4 papers here."
authors: ["Juliette Millet", "Ewan Dunbar"]
year: 2020
venue: "arXiv:2005.03418"
source: "https://arxiv.org/abs/2005.03418"
doi: "10.48550/arxiv.2005.03418"
arxiv: "2005.03418"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/millet-2020-perceptimatic-english-benchmark-speech-perception/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/millet-2020-perceptimatic-english-benchmark-speech-perception/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/7/) / The Perceptimatic English Benchmark for Speech Perception Models

# The Perceptimatic English Benchmark for Speech Perception Models

**Authors:** Juliette Millet and Ewan Dunbar · **Year:** 2020 · **Venue:** arXiv:2005.03418 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2005.03418) · [PDF](https://arxiv.org/pdf/2005.03418) · [DOI 10.48550/arxiv.2005.03418](https://doi.org/10.48550/arxiv.2005.03418) · [arXiv 2005.03418](https://arxiv.org/abs/2005.03418) · [OpenAlex](https://openalex.org/W3023172065)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [4](https://md.research.elamur.ai/free/en/papers/millet-2020-perceptimatic-english-benchmark-speech-perception/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Speech recognizers process natural recorded speech, so they could serve as a starting point for models of human speech perception. However, no one had systematically measured how their behaviour differs from that of listeners. The authors built the Perceptimatic English Benchmark (PEB) to fill this gap. It is an open set of ABX phone-discrimination items cut from read audiobook speech, which matches the kind of data typical acoustic models are trained on. Each item compares two three-phone snippets that differ only in the middle phone, and the benchmark covers English and French contrasts. American English listeners judged every item online. To test a model, they compute a graded discriminability score from dynamic-time-warping distances between the model's representations and relate it to human accuracy with a binomial regression. Of seven models tested, a multilingual phone model and an unsupervised short-event model (DPGMM) tracked listeners well. DeepSpeech solved the task accurately but matched human judgments poorly, which suggests it is overly tuned to English. The resource gives researchers a ready human norm for perceptual distance.

## Key points
- Responses come from 91 American English listeners recruited on Amazon Mechanical Turk and tested with LMEDS. Another 65 people were excluded for failing catch trials or not finishing.
- The stimuli comprise 5202 triplets (2214 English) and 461 centre-phone contrasts (212 English, 249 French) across 201 contexts. They were drawn from the LibriVox subset used in the Zero Resource Speech Challenge 2017.
- In each triplet, A and B come from one speaker and X from a different speaker. The benchmark uses 15 English and 18 French speakers, and each triplet appears in four orderings.
- Listeners answered on a six-point confidence scale, and the dataset also includes a binarized version. The reported analyses use the binary answers.
- Model predictions use δ = d(Other, X) − d(Target, X), where d is a DTW distance with either cosine or symmetrised KL as the frame-level measure. The authors recommend the off-the-shelf multilingual model (via shennong) for estimating perceptual distance.

## Interesting ideas
- Instead of scoring a model by task accuracy, they turn its internal distances into a graded margin and test whether that margin predicts human behaviour item by item. This rewards a human-like pattern of difficulty, not just correct answers.
- DeepSpeech scored well on the task but was a poor predictor of humans, and splitting results into native (English) versus foreign (French) items exposed its over-specialization.
- Each item is a minimal triplet in which only the centre unit changes while the surrounding context is held fixed and hand-checked, so any response difference can be traced to one element.
- Easy catch trials, such as telling "cat" from "dog", were mixed in to identify inattentive participants and drop them.
- Every stimulus set was shown in all four A/B/X orderings, and speaker combinations were counterbalanced so that they never revealed the answer.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Parser or reader confidence margins could be regressed against disagreement among witness parsers or LLM teachers, revealing whether Mova finds hard the same sentences that are genuinely ambiguous.
- **I2** (learn): Evaluations could pair in-domain test sets with deliberately foreign or out-of-style text as a control, flagging components that only memorized the training register.
- **I3** (tree): Build minimal-pair sentence sets that differ in one word or attachment site to test exactly which token flips the UD analysis, making parser errors explainable.
- **I4** (learn): Planting trivially checkable items among silver-data or LLM-teacher annotation batches would let unreliable annotators or teacher tiers be filtered automatically.
- **I5** (answer): School-test and fable QA evaluation could permute answer options and surface cues so that a correct answer cannot come from position or formatting bias.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Laboratoire de Linguistique Formelle](https://md.research.elamur.ai/free/en/labs/laboratoire-de-linguistique-formelle/) — Research institute, Paris, France · 10 works · 481 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 798 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 463 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 213 works

🧭 Topics (2): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works

## Cite

Cite as: Juliette Millet and Ewan Dunbar. 2020. The Perceptimatic English Benchmark for Speech Perception Models. arXiv:2005.03418.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{millet2020perceptimatic,
  title = {{The Perceptimatic English Benchmark for Speech Perception Models}},
  author = {Juliette Millet and Ewan Dunbar},
  year = {2020},
  doi = {10.48550/arxiv.2005.03418},
  eprint = {2005.03418},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2005.03418}
}
```

- [← previous Shallow Discourse Parsing for Under-Resourced Languages: Combining Machine Translation and Annotation Projection](https://md.research.elamur.ai/free/en/papers/gathje-2020-shallow-discourse-parsing-under-resourced/)
- [next → An Empirical Study of Contextual Data Augmentation for Japanese Zero Anaphora Resolution](https://md.research.elamur.ai/free/en/papers/konno-2020-empirical-study-contextual-data-augmentation/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/7/)

## References

27 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Chen, H., Leung, C.-C., Xie, L., Ma, B., & Li, H. (2015). Parallel inference of Dirichlet process Gaussian mixture models for unsupervised acoustic modeling: A feasibility study. In INTERSPEECH-16. [find](https://scholar.google.com/scholar?q=Chen%2C+H%2E%2C+Leung%2C+C%2E%2DC%2E%2C+Xie%2C+L%2E%2C+Ma%2C+B%2E%2C+%26+Li%2C+H%2E+%282015%29%2E+Parallel+inference+of+Dirichlet+process+Gaussian+mixture+models+for+unsupervised+acoustic+modeling%3A+A+feasibility+study%2E+In+INTERSPEECH%2D16%2E)
2. Cieri, C., Miller, D., & Walker, K. (2004). [The fisher corpus: a resource for the next generations of speech-to-text](https://md.research.elamur.ai/free/en/papers/cieri-2004-fisher-corpus-resource-next-generations/). In Lrec.
3. Dunbar, E., Cao, X. N., Benjumea, J., Karadayi, J., Bernard, M., Besacier, L., . . . Dupoux, E. (2017). [find](https://scholar.google.com/scholar?q=Dunbar%2C+E%2E%2C+Cao%2C+X%2E+N%2E%2C+Benjumea%2C+J%2E%2C+Karadayi%2C+J%2E%2C+Bernard%2C+M%2E%2C+Besacier%2C+L%2E%2C+%2E+%2E+%2E+Dupoux%2C+E%2E+%282017%29%2E)
4. The Zero Resource Speech Challenge 2017. In 2017 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) (pp. 323–330). [find](https://scholar.google.com/scholar?q=The+Zero+Resource+Speech+Challenge+2017%2E+In+2017+IEEE+Workshop+on+Automatic+Speech+Recognition+and+Understanding+%28ASRU%29+%28pp%2E+323%E2%80%93330%29%2E)
5. Dupoux, E. (2018). Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language-learner. Cognition, 173, 43–59. [find](https://scholar.google.com/scholar?q=Dupoux%2C+E%2E+%282018%29%2E+Cognitive+science+in+the+era+of+artificial+intelligence%3A+A+roadmap+for+reverse%2Dengineering+the+infant+language%2Dlearner%2E+Cognition%2C+173%2C+43%E2%80%9359%2E)
6. Elman, J., & McClelland, J. (2015). Exploiting the lawful variability in the speech wave. In J. Perkell & D. Klatt (Eds.), (Vol. 335, pp. 71–90). Hillsdale, NJ: Erlbaum. [find](https://scholar.google.com/scholar?q=Elman%2C+J%2E%2C+%26+McClelland%2C+J%2E+%282015%29%2E+Exploiting+the+lawful+variability+in+the+speech+wave%2E+In+J%2E+Perkell+%26+D%2E+Klatt+%28Eds%2E%29%2C+%28Vol%2E+335%2C+pp%2E+71%E2%80%9390%29%2E+Hillsdale%2C+NJ%3A+Erlbaum%2E)
7. Feldman, N. H., & Griffiths, T. L. (2007). A rational account of the perceptual magnet effect. In Proceedings of the annual meeting of the cognitive science society (Vol. 29). [find](https://scholar.google.com/scholar?q=Feldman%2C+N%2E+H%2E%2C+%26+Griffiths%2C+T%2E+L%2E+%282007%29%2E+A+rational+account+of+the+perceptual+magnet+effect%2E+In+Proceedings+of+the+annual+meeting+of+the+cognitive+science+society+%28Vol%2E+29%29%2E)
8. Fer, R., Matejka, P., Grezl, F., Plchot, O., Vesely, K., & Cernocky, J. H. (2017). Multilingually trained bottleneck features in spoken language recognition. Computer Speech and Language, 46(Supplement C), 252 - 267. [↗](https://doi.org/10.1016/j.csl.2017.06.008)
9. Godfrey, J. J., Holliman, E. C., & McDaniel, J. (1992). Switchboard: Telephone speech corpus for research and development. In [proceedings] icassp-92: 1992 ieee international conference on acoustics, speech, and signal processing (Vol. 1, pp. 517–520). [find](https://scholar.google.com/scholar?q=Godfrey%2C+J%2E+J%2E%2C+Holliman%2C+E%2E+C%2E%2C+%26+McDaniel%2C+J%2E+%281992%29%2E+Switchboard%3A+Telephone+speech+corpus+for+research+and+development%2E+In+%5Bproceedings%5D+icassp%2D92%3A+1992+ieee+international+conference+on+acoustics%2C+)
10. Guenther, F., & Gjaja, M. (1996). The perceptual magnet effect as an emergent property of neural map formation. The Journal of the Acoustical Society of America, 100(2), 1111–1121. [↗](https://doi.org/10.1121/1.416296)
11. Hannun, A., Case, C., Casper, J., Catanzaro, B., Diamos, G., Elsen, E., . . . others (2014). [Deep speech: Scaling up end-to-end speech recognition](https://md.research.elamur.ai/free/en/papers/hannun-2014-deep-speech-scaling-up-end/). arXiv preprint arXiv:1412.5567. [↗](https://arxiv.org/abs/1412.5567)
12. Jurov, N. (2019). Phonetics or Phonology? Modelling Non-Native Perception. Unpublished master’s thesis, Universit Paris Diderot, Paris, France. [find](https://scholar.google.com/scholar?q=Jurov%2C+N%2E+%282019%29%2E+Phonetics+or+Phonology%3F+Modelling+Non%2DNative+Perception%2E+Unpublished+master%E2%80%99s+thesis%2C+Universit+Paris+Diderot%2C+Paris%2C+France%2E)
13. Kleinschmidt, D., & Jaeger, T. F. (2015). Robust speech perception: Recognize the familiar, generalize to the similar, and adapt to the novel. Psychological Review, 122(2), 148–203. [↗](https://doi.org/10.1037/a0038695)
14. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., . . . others (2019). [Natural questions: a benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Linguistics, 7, 453–466. [↗](https://doi.org/10.1162/tacl_a_00276)
15. Mahrt, T. (2016). LMEDS: Language markup and experimental design software. [find](https://scholar.google.com/scholar?q=Mahrt%2C+T%2E+%282016%29%2E+LMEDS%3A+Language+markup+and+experimental+design+software%2E)
16. McClelland, J., & Elman, J. (1986). Interactive processes in speech perception: The TRACE model. Cognitive Psychology, 18, 1–86. [find](https://scholar.google.com/scholar?q=McClelland%2C+J%2E%2C+%26+Elman%2C+J%2E+%281986%29%2E+Interactive+processes+in+speech+perception%3A+The+TRACE+model%2E+Cognitive+Psychology%2C+18%2C+1%E2%80%9386%2E)
17. Millet, J., Jurov, N., & Dunbar, E. (2019). Comparing unsupervised speech learning directly to human performance in speech perception.. [↗](https://doi.org/10.31234/osf.io/ake47)
18. Norris, D., & McQueen, J. (2008). Shortlist B: a Bayesian model of continuous speech recognition. Psychological Review, 115(2), 357–395. [↗](https://doi.org/10.1037/0033-295x.115.2.357)
19. Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015). Librispeech: an asr corpus based on public domain audio books. In 2015 ieee international conference on acoustics, speech and signal processing (icassp) (pp. 5206–5210). [find](https://scholar.google.com/scholar?q=Panayotov%2C+V%2E%2C+Chen%2C+G%2E%2C+Povey%2C+D%2E%2C+%26+Khudanpur%2C+S%2E+%282015%29%2E+Librispeech%3A+an+asr+corpus+based+on+public+domain+audio+books%2E+In+2015+ieee+international+conference+on+acoustics%2C+speech+and+signal+process)
20. Parrot, M., Millet, J., & Dunbar, E. (2019). Independent and automatic evaluation of acoustic-to-articulatory inversion models. arXiv, arXiv–1911. [↗](https://doi.org/10.48550/arxiv.1911.06573)
21. Povey, D., Ghoshal, A., Boulianne, G., Burget, L., Glembek, O., Goel, N., . . . others (2011). The Kaldi speech recognition toolkit. In IEEE 2011 Workshop on Automatic Speech Recognition and Understanding (ASRU). [find](https://scholar.google.com/scholar?q=Povey%2C+D%2E%2C+Ghoshal%2C+A%2E%2C+Boulianne%2C+G%2E%2C+Burget%2C+L%2E%2C+Glembek%2C+O%2E%2C+Goel%2C+N%2E%2C+%2E+%2E+%2E+others+%282011%29%2E+The+Kaldi+speech+recognition+toolkit%2E+In+IEEE+2011+Workshop+on+Automatic+Speech+Recognition+and+Understan)
22. Riochet, R., Castro, M. Y., Bernard, M., Lerer, A., Fergus, R., Izard, V., & Dupoux, E. (2018). Intphys: A framework and benchmark for visual intuitive physics reasoning. arXiv preprint arXiv:1803.07616. [↗](https://arxiv.org/abs/1803.07616)
23. Scharenborg, O., Norris, D., ten Bosch, L., & McQueen, J. (2005). How should a speech recognizer work? Cognitive Science, 29, 867–918. [↗](https://doi.org/10.1207/s15516709cog0000_37)
24. Schatz, T., Bach, F., & Dupoux, E. (2017). ASR systems as models of phonetic category perception in adults. In Proceedings of the 39th Annual CogSci Meeting. [↗](https://doi.org/10.17605/osf.io/57d8x)
25. Schatz, T., & Feldman, N. (2018). Neural network vs. HMM speech recognition systems as models of human cross-linguistic phonetic perception. In Proceedings of the conference on cognitive computational neuroscience (pp. 1–4). [find](https://scholar.google.com/scholar?q=Schatz%2C+T%2E%2C+%26+Feldman%2C+N%2E+%282018%29%2E+Neural+network+vs%2E+HMM+speech+recognition+systems+as+models+of+human+cross%2Dlinguistic+phonetic+perception%2E+In+Proceedings+of+the+conference+on+cognitive+computational)
26. Tiede, M., Espy-Wilson, C. Y., Goldenberg, D., Mitra, V., Nam, H., & Sivaraman, G. (2017). Quantifying kinematic aspects of reduction in a contrasting rate production task. The Journal of the Acoustical Society of America, 141(5), 3580-3580. Retrieved from https://doi.org/10.1121/1.4987629 doi: 10.1121/1.4987629 [↗](https://doi.org/10.1121/1.4987629)
27. Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S.-F., & Bowman, S. (2019). Blimp: A benchmark of linguistic minimal pairs for english. arXiv preprint arXiv:1912.00582. [↗](https://arxiv.org/abs/1912.00582)

*Updated 24 October 2026*