---
title: "How AI Fails — paper by Ghimire (2025) — summary and ideas"
description: "The paper asks whether automated toxicity classifiers penalise writers because of their dialect rather than their message, and… 22 references."
authors: ["Subhojit Ghimire"]
year: 2025
venue: "arXiv:2511.06676"
source: "https://arxiv.org/abs/2511.06676"
doi: "10.48550/arxiv.2511.06676"
arxiv: "2511.06676"
licence: "CC BY-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ghimire-2025-ai-fails-interactive-pedagogical-tool/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ghimire-2025-ai-fails-interactive-pedagogical-tool/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/17/) / How AI Fails: An Interactive Pedagogical Tool for Demonstrating Dialectal Bias in Automated Toxicity Models

# How AI Fails: An Interactive Pedagogical Tool for Demonstrating Dialectal Bias in Automated Toxicity Models

**Authors:** Subhojit Ghimire · **Year:** 2025 · **Venue:** arXiv:2511.06676 · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2511.06676) · [PDF](https://arxiv.org/pdf/2511.06676) · [DOI 10.48550/arxiv.2511.06676](https://doi.org/10.48550/arxiv.2511.06676) · [arXiv 2511.06676](https://arxiv.org/abs/2511.06676) · [OpenAlex](https://openalex.org/W4416157935)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper asks whether automated toxicity classifiers penalise writers because of their dialect rather than their message, and how to make that harm understandable to non-experts. The author audits unitary/toxic-bert, a bert-base-uncased model fine-tuned on Jigsaw Toxic Comment data. The audit uses two balanced corpora of 10,000 tweets each, one in African-American English (AAE) and one in Standard American English (SAE), drawn from the TwitterAAE corpus by Slanglab. The tweets were selected by high demographic-model confidence and cleaned of URLs, handles and similar noise.

On average the model gives AAE text about 1.8× higher toxicity and 8.8× higher identity-attack scores. False positive rates diverge across classification thresholds.

The second contribution is a public web tool, the Dialectal Bias Analyser. It shows all six label scores and lets users move a decision threshold themselves. The point it makes is that the real harm comes from a human-chosen cut-off, which looks neutral but still silences one dialect more than the other. The work matters as a bridge between fairness research and public AI literacy.

## Key points
- Model audited: unitary/toxic-bert, run through a transformers pipeline with sigmoid outputs, so each of its six categories (identity attack, insult, threat, obscene, severe toxicity and toxicity) receives its own independent probability.
- Data: the 12 GB twitteraae_all file was streamed with Pandas in one-million-row chunks. Malformed rows were dropped, and only tweets with demographic probability above 80% were kept, giving 10k AAE and 10k SAE samples.
- Results: AAE scored about 1.8× more toxic and about 8.8× higher on identity hate than SAE. False positive rate was plotted against the threshold, under the assumption that all 20,000 tweets are benign.
- Tool: a Flask backend with a single /api/predict/ endpoint, plus a React/Babel front end. It includes preset minimal pairs such as zero copula ("She at the library studying") and double negation.
- Stated limitations:
  - Only one model was tested; its successor, detoxify, was not checked.
  - Dialect labels come from another model, not from human annotation.
  - FPR is only approximate, because some tweets may genuinely be toxic.

## Interesting ideas
- Present the same meaning in two dialects as a minimal pair, so that any change in output can only come from surface form; this cleanly separates "what is said" from "how it is said".
- Keep the model's score strictly apart from the decision threshold, and show error rates for each group across the whole threshold range instead of at a single cut-off; this exposes the policy choice hidden in "neutral" defaults.
- Use simple ratio metrics (mean score of group A divided by group B, per output label) as a quick disparity check; they are easy to explain and reveal which label carries most of the bias.
- Build balanced corpora from a huge, probabilistically labelled stream by keeping only high-confidence items and capping each side at equal size.
- Show every intermediate score and let users move the decision boundary themselves; interaction makes the cause of an outcome visible, which a static report does not.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tree): Build AAE/SAE minimal-pair test sets (zero copula, negative concord, ain't) and check that the parser yields equivalent UD trees, for example a predicate without an overt cop node mapped to the same frame.
- **I2** (learn): When accepting silver parses or repair rules by confidence, report acceptance and error curves per register or dialect, so a global threshold does not quietly discard non-standard text.
- **I3** (answer): Compare answer accuracy and confidence on paired standard and dialectal versions of school tests and fables, per question type, to spot where the reader breaks.
- **I4** (learn): Select silver sentences where several witness parsers agree strongly, while balancing sizes across genres, so training is not dominated by the majority style.
- **I5** (frames): Expose speech-act or irony scores alongside the chosen label in an inspector, so a developer can see which cue pushed a dialectal sentence over a boundary.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 543 works · [🤝 Political Science and International Relations](https://md.research.elamur.ai/free/en/shelves/social-sciences/political-science-and-international-relations/) · 182 works

🧭 Topics (1): [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 517 works

## Cite

Cite as: Subhojit Ghimire. 2025. How AI Fails: An Interactive Pedagogical Tool for Demonstrating Dialectal Bias in Automated Toxicity Models. arXiv:2511.06676.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ghimire2025how,
  title = {{How AI Fails: An Interactive Pedagogical Tool for Demonstrating Dialectal Bias in Automated Toxicity Models}},
  author = {Subhojit Ghimire},
  year = {2025},
  doi = {10.48550/arxiv.2511.06676},
  eprint = {2511.06676},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2511.06676}
}
```

- [← previous Hierarchical Section Matching Prediction (HSMP) BERT for Fine-Grained Extraction of Structured Data from Hebrew Free-Text Radiology Reports in Crohn's Disease](https://md.research.elamur.ai/free/en/papers/badash-2025-hierarchical-section-matching-prediction-hsmp/)
- [next → Huaputong: Large Language Model for Genealogical Question-answering with Knowledge Reasoning](https://md.research.elamur.ai/free/en/papers/wu-2025-huaputong-large-language/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/17/)

## References

22 references, 2 of them with a card of ours. ↗ leads to the work.

1. V. U. Gongane, M. V. Munot, and A. D. Anuse, “Detection and moderation of detrimental content on social media platforms: current status and future directions,” Social Network Analysis and Mining, vol. 12, no. 1, p. 129, Sep 2022. [Online]. Available: https://doi.org/10.1007/s13278-022-00951-3 [↗](https://doi.org/10.1007/s13278-022-00951-3)
2. J. Burrell, “How the machine ‘thinks’: Understanding opacity in machine learning algorithms,” Big Data & Society, vol. 3, no. 1, p. 2053951715622512, 2016. [Online]. Available: https://doi.org/10.1177/2053951715622512 [↗](https://doi.org/10.1177/2053951715622512)
3. S. U. Noble, Algorithms of Oppression, ser. How Search Engines Reinforce Racism. New York, USA: New York University Press, 2018. [Online]. Available: https://doi.org/10.18574/nyu/9781479833641.001. 0001 [↗](https://doi.org/10.18574/nyu/9781479833641.001)
4. S. L. Blodgett, L. Green, and B. O’Connor, “[Demographic dialectal variation in social media: A case study of African-American English](https://md.research.elamur.ai/free/en/papers/blodgett-2016-demographic-dialectal-variation-social-media/),” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, J. Su, K. Duh, and X. Carreras, Eds. Austin, Texas: Association for Computational Linguistics, Nov. 2016, pp. 1119–1130. [Online]. Available: https://aclanthology.org/D16-1120/ [↗](https://aclanthology.org/D16-1120/)
5. J. F. Gomez, C. Machado, L. M. Paes, and F. Calmon, “Algorithmic arbitrariness in content moderation,” in Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, ser. FAccT ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 2234–2253. [Online]. Available: https://doi.org/10. 1145/3630106.3659036 [↗](https://doi.org/10)
6. “unitary/toxic-bert,” [https://huggingface.co/unitary/toxic-bert/](https://huggingface.co/unitary/toxic-bert/), accessed: 2026-02. [↗](https://huggingface.co/unitary/toxic-bert/)
7. “Twitteraae dataset,” [https://slanglab.cs.umass.edu/TwitterAAE/](https://slanglab.cs.umass.edu/TwitterAAE/), accessed: 2026-02. [↗](https://slanglab.cs.umass.edu/TwitterAAE/)
8. M. Sap, D. Card, S. Gabriel, Y. Choi, and N. A. Smith, “[The risk of racial bias in hate speech detection](https://md.research.elamur.ai/free/en/papers/sap-2019-risk-racial-bias-hate-speech/),” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. Màrquez, Eds. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 1668–1678. [Online]. Available: https://aclanthology.org/P19-1163/ [↗](https://aclanthology.org/P19-1163/)
9. A. Parikh, S. Das, and A. Feragen, “Investigating label bias and representational sources of age-related disparities in medical segmentation,” in 2026 IEEE 23rd International Symposium on Biomedical Imaging (ISBI), 2026, pp. 1–5. [Online]. Available: https://doi.org/10.1109/ISBI61048.2026.11515928 [↗](https://doi.org/10.1109/isbi61048.2026.11515928)
10. I. Tenney, J. Wexler, J. Bastings, T. Bolukbasi, A. Coenen, S. Gehrmann, E. Jiang, M. Pushkarna, C. Radebaugh, E. Reif, and A. Yuan, “The language interpretability tool: Extensible, interactive visualizations and analysis for NLP models,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Q. Liu and D. Schlangen, Eds. Online: Association for Computational Linguistics, Oct. 2020, pp. 107–118. [Online]. Available: https://aclanthology.org/2020.emnlp-demos.15/ [↗](https://aclanthology.org/2020.emnlp-demos.15/)
11. “biaslyze,” [https://biaslyze.org/](https://biaslyze.org/), accessed: 2026-02. [↗](https://biaslyze.org/)
12. S. Kabir, L. Li, and T. Zhang, “Stile: Exploring and debugging social biases in pre-trained text representations,” in Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, ser. CHI ’24. New York, NY, USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/10.1145/3613904.3642111 [↗](https://doi.org/10.1145/3613904.3642111)
13. H. Viswanath and T. Zhang, “Fairpy: A toolkit for evaluation of prediction biases and their mitigation in large language models,” 2025. [Online]. Available: https://arxiv.org/abs/2302.05508 [↗](https://arxiv.org/abs/2302.05508)
14. D. Raz, C. Bintz, V. Guetler, A. Tam, M. Katell, D. Dailey, B. Herman, P. M. Krafft, and M. Young, “Face mis-id: An interactive pedagogical tool demonstrating disparate accuracy rates in facial recognition,” in Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, ser. AIES ’21. New York, NY, USA: Association for Computing Machinery, 2021, p. 895–904. [Online]. Available: https://doi.org/10.1145/3461702.3462627 [↗](https://doi.org/10.1145/3461702.3462627)
15. “How normal am i,” [https://www.hownormalami.eu/](https://www.hownormalami.eu/), accessed: 2026-02. [↗](https://www.hownormalami.eu/)
16. “evaluate-measurement/toxicity,” [https://huggingface.co/spaces/](https://huggingface.co/spaces/) evaluate-measurement/toxicity/, accessed: 2026-02. [↗](https://huggingface.co/spaces/)
17. A. Field, S. L. Blodgett, Z. Waseem, and Y. Tsvetkov, “A survey of race, racism, and anti-racism in NLP,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), C. Zong, F. Xia, W. Li, and R. Navigli, Eds. Online: Association for Computational Linguistics, Aug. 2021, pp. 1905–1925. [Online]. Available: https://aclanthology.org/2021.acl-long.149/ [↗](https://aclanthology.org/2021.acl-long.149/)
18. Z. Yang, Y. Maricar, M. Davari, N. Grenon-Godbout, and R. Rabbany, “Toxbuster: In-game chat toxicity buster with bert,” 2023. [Online]. Available: https://arxiv.org/abs/2305.12542 [↗](https://arxiv.org/abs/2305.12542)
19. “bert-base-uncased,” [https://huggingface.co/google-bert/](https://huggingface.co/google-bert/) bert-base-uncased, accessed: 2026-02. [↗](https://huggingface.co/google-bert/)
20. “Jigsaw toxic comment classification challenge,” [https://huggingface.co/](https://huggingface.co/) datasets/thesofakillers/jigsaw-toxic-comment-classification-challenge, accessed: 2026-02. [↗](https://huggingface.co/)
21. “Dialectal bias analyser repository,” [https://github.com/SubhojitGhimire/](https://github.com/SubhojitGhimire/) Dialectal Bias Analyser, accessed: 2026-02. [↗](https://github.com/SubhojitGhimire/)
22. “Dialectal bias analyser: Hugging face space,” [https://huggingface.co/](https://huggingface.co/) spaces/SubhojitGhimire/Dialect Biasness Analyser, accessed: 2026-02. [↗](https://huggingface.co/)

*Updated 30 October 2026*