---
title: "How Well Do Commodity Text-to-Speech Systems Evade"
description: "This paper tests whether that still holds for free, scriptable TTS services of the kind used in mass vishing campaigns. 13 references."
authors: ["Anatoly Marchenko"]
year: 2026
venue: "Proceedings of the Second International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security"
source: "https://aclanthology.org/2026.nlpaics-1.18/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/marchenko-2026-how-well-do-commodity-text/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/marchenko-2026-how-well-do-commodity-text/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/32/) / How Well Do Commodity Text-to-Speech Systems Evade Acoustic Perturbation Detection? A Multi-Engine Evaluation Across 21 Languages

# How Well Do Commodity Text-to-Speech Systems Evade Acoustic Perturbation Detection? A Multi-Engine Evaluation Across 21 Languages

**Authors:** Anatoly Marchenko · **Year:** 2026 · **Venue:** [Proceedings of the Second International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security](https://md.research.elamur.ai/free/en/venues/nlpaics/) ([International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security 2026](https://md.research.elamur.ai/free/en/venues/nlpaics/2026/)) · **Publisher:** [Department of Languages and Information Systems, University of Alicante](https://md.research.elamur.ai/free/en/publishers/department-of-languages-and-information-systems-university-of-alicante/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.nlpaics-1.18/) · [PDF](https://aclanthology.org/2026.nlpaics-1.18.pdf)

## Summary
Many voice-deepfake detectors rely on jitter, shimmer and harmonics-to-noise ratio (HNR). The assumption is that synthetic voices lack the small biological irregularities of human vocal fold vibration. This paper tests whether that still holds for free, scriptable TTS services of the kind used in mass vishing campaigns.

The author built a natural-speech reference from 29 public emotion corpora (24 languages, 11 families, 35,091 utterances). Emotional speech was chosen because phone fraud exploits urgency and distress. The comparison set is 1,850 clips from Google TTS, Microsoft Edge TTS and the macOS system voice. All three features were extracted with Parselmouth. Separability was assessed in two ways: Mann-Whitney U tests with rank-biserial effect sizes, and three classifiers trained on repeated balanced subsamples with cross-validation.

gTTS remains easy to catch because its output is unnaturally clean. Edge TTS, however, approaches human perturbation profiles, and no classifier reaches F1 above 0.78 against it. For anti-fraud and telephony teams, these features are still useful as one signal among several, but they are not a sufficient defense on their own, especially across languages.

## Key points
- **Data:** gTTS covers 12 languages (159 samples), macOS 9 languages (635), and Edge TTS 21 languages (1,056). The classifiers are logistic regression, SVM-RBF and Random Forest, evaluated with 10 subsamples × 5-fold CV.
- **Classifier results:** Against Edge TTS, the strongest model is SVM-RBF (F1 = 0.78, AUROC = 0.85). This is well short of the roughly 0.90 the author considers necessary for a standalone detector.
- **Effect sizes:** Moving from gTTS to Edge TTS, effect sizes fall by 2.1× (shimmer), 4.3× (jitter) and 7.4× (HNR). The largest |r| drops from 0.75 (gTTS) to 0.56 (macOS) to 0.36 (Edge TTS).
- **Ablation:** Against Edge TTS, single features score F1 = 0.52 (jitter), 0.57 (shimmer) and 0.60 (HNR). Combining all three reaches only 0.75 with Random Forest.
- **Cross-language results:** In 14 of the 21 languages, Edge TTS and natural speech cannot be told apart by jitter once Bonferroni correction is applied. A neutral-only baseline check confirms that the emotional corpora do not explain the result.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 480 works

## Cite

Cite as: Anatoly Marchenko. 2026. How Well Do Commodity Text-to-Speech Systems Evade Acoustic Perturbation Detection? A Multi-Engine Evaluation Across 21 Languages. Proceedings of the Second International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{marchenko2026how,
  title = {{How Well Do Commodity Text-to-Speech Systems Evade Acoustic Perturbation Detection? A Multi-Engine Evaluation Across 21 Languages}},
  author = {Anatoly Marchenko},
  booktitle = {Proceedings of the Second International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security},
  year = {2026},
  url = {https://aclanthology.org/2026.nlpaics-1.18/}
}
```

- [← previous How to Train a Real-World Silicon Concierge? Internalizing Complex Business Workflow to Only OneModel](https://md.research.elamur.ai/free/en/papers/tong-2026-how-train-real-world-silicon/)
- [next → HowToNarrate: A General-Domain Benchmark for Synchronized Video Narration with External Knowledge](https://md.research.elamur.ai/free/en/papers/wang-2026-howtonarrate-general-domain-benchmark-synchronized/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/32/)

## References

13 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. We tested three commodity TTS engines against perturbation-based detection across 21 languages (1,850 synthetic vs. 35,091 natural samples from 29 corpora). Three classifiers and ablation analysis all point the same way: effect sizes shrink 2–7× from older to newer TTS, Edge TTS reaches only small e… [find](https://scholar.google.com/scholar?q=We+tested+three+commodity+TTS+engines+against+perturbation%2Dbased+detection+across+21+languages+%281%2C850+synthetic+vs%2E+35%2C091+natural+samples+from+29+corpora%29%2E+Three+classifiers+and+ablation+analysis+all)
2. Yannick Jadoul, Bill Thompson, and Bart de Boer. 2018. Introducing Parselmouth: A Python interface to Praat. Journal of Phonetics, 71:1–15. [find](https://scholar.google.com/scholar?q=Yannick+Jadoul%2C+Bill+Thompson%2C+and+Bart+de+Boer%2E+2018%2E+Introducing+Parselmouth%3A+A+Python+interface+to+Praat%2E+Journal+of+Phonetics%2C+71%3A1%E2%80%9315%2E)
3. Aurosweta Mahapatra, Ismail Rasim Ulgen, Abinay Reddy Naini, Carlos Busso, and Berrak Sisman. 2025. Can emotion fool anti-spoofing? In Proc. Interspeech. [find](https://scholar.google.com/scholar?q=Aurosweta+Mahapatra%2C+Ismail+Rasim+Ulgen%2C+Abinay+Reddy+Naini%2C+Carlos+Busso%2C+and+Berrak+Sisman%2E+2025%2E+Can+emotion+fool+anti%2Dspoofing%3F+In+Proc%2E+Interspeech%2E)
4. Nicolas M. Müller, Pavel Czempin, Franziska Diekmann, Adam Froghyar, and Konstantin Böttinger. 2022. Does audio deepfake detection generalize? In Proc. Interspeech, pages 2783–2787. [find](https://scholar.google.com/scholar?q=Nicolas+M%2E+M%C3%BCller%2C+Pavel+Czempin%2C+Franziska+Diekmann%2C+Adam+Froghyar%2C+and+Konstantin+B%C3%B6ttinger%2E+2022%2E+Does+audio+deepfake+detection+generalize%3F+In+Proc%2E+Interspeech%2C+pages+2783%E2%80%932787%2E)
5. Hemlata Tak, Jose Patino, Massimiliano Todisco, Andreas Nautsch, Nicholas Evans, and Anthony Larcher. 2021. End-to-end anti-spoofing with RawNet2. In Proc. ICASSP, pages 6369–6373. [find](https://scholar.google.com/scholar?q=Hemlata+Tak%2C+Jose+Patino%2C+Massimiliano+Todisco%2C+Andreas+Nautsch%2C+Nicholas+Evans%2C+and+Anthony+Larcher%2E+2021%2E+End%2Dto%2Dend+anti%2Dspoofing+with+RawNet2%2E+In+Proc%2E+ICASSP%2C+pages+6369%E2%80%936373%2E)
6. Ingo R. Titze. 1994. Principles of Voice Production. Prentice Hall. [find](https://scholar.google.com/scholar?q=Ingo+R%2E+Titze%2E+1994%2E+Principles+of+Voice+Production%2E+Prentice+Hall%2E)
7. Xin Wang, Héctor Delgado, Hemlata Tak, Jee-weon Jung, Hye-jin Shim, Massimiliano Todisco, Ivan Kukanov, Xuechen Liu, Md Sahidullah, Tomi H. Kinnunen, Nicholas Evans, Kong Aik Lee, and Junichi Yamagishi. 2024. ASVspoof 5: Crowdsourced speech data, deepfakes, and adversarial attacks at scale. In Proc. [find](https://scholar.google.com/scholar?q=Xin+Wang%2C+H%C3%A9ctor+Delgado%2C+Hemlata+Tak%2C+Jee%2Dweon+Jung%2C+Hye%2Djin+Shim%2C+Massimiliano+Todisco%2C+Ivan+Kukanov%2C+Xuechen+Liu%2C+Md+Sahidullah%2C+Tomi+H%2E+Kinnunen%2C+Nicholas+Evans%2C+Kong+Aik+Lee%2C+and+Junichi+Yamagish)
8. ASVspoof 2024 Workshop, pages 1–8. [find](https://scholar.google.com/scholar?q=ASVspoof+2024+Workshop%2C+pages+1%E2%80%938%2E)
9. Kevin Warren, Daniel Olszewski, Seth Layton, Kevin Butler, Carrie Gates, and Patrick Traynor. 2025. Pitch imperfect: Detecting audio deepfakes through acoustic prosodic analysis. arXiv preprint arXiv:2502.14726. — [arXiv:2502.14726](https://arxiv.org/abs/2502.14726) [find](https://scholar.google.com/scholar?q=Kevin+Warren%2C+Daniel+Olszewski%2C+Seth+Layton%2C+Kevin+Butler%2C+Carrie+Gates%2C+and+Patrick+Traynor%2E+2025%2E+Pitch+imperfect%3A+Detecting+audio+deepfakes+through+acoustic+prosodic+analysis%2E+arXiv+preprint+arXiv%3A)
10. Junichi Yamagishi, Xin Wang, Massimiliano Todisco, Md Sahidullah, Jose Patino, Andreas Nautsch, Xuechen Liu, Kong Aik Lee, Tomi Kinnunen, Nicholas Evans, and Héctor Delgado. 2021. [find](https://scholar.google.com/scholar?q=Junichi+Yamagishi%2C+Xin+Wang%2C+Massimiliano+Todisco%2C+Md+Sahidullah%2C+Jose+Patino%2C+Andreas+Nautsch%2C+Xuechen+Liu%2C+Kong+Aik+Lee%2C+Tomi+Kinnunen%2C+Nicholas+Evans%2C+and+H%C3%A9ctor+Delgado%2E+2021%2E)
11. ASVspoof 2021: Accelerating progress in spoofed and deepfake speech detection. In Proc. [find](https://scholar.google.com/scholar?q=ASVspoof+2021%3A+Accelerating+progress+in+spoofed+and+deepfake+speech+detection%2E+In+Proc%2E)
12. ASVspoof 2021 Workshop. [find](https://scholar.google.com/scholar?q=ASVspoof+2021+Workshop%2E)
13. Tianle Yang, Chengzhe Sun, Siwei Lyu, and Phil Rose. 2026. Forensic deepfake audio detection using segmental speech features. Forensic Science International, 379:112768. [find](https://scholar.google.com/scholar?q=Tianle+Yang%2C+Chengzhe+Sun%2C+Siwei+Lyu%2C+and+Phil+Rose%2E+2026%2E+Forensic+deepfake+audio+detection+using+segmental+speech+features%2E+Forensic+Science+International%2C+379%3A112768%2E)

*Updated 1 November 2026*