---
title: "The Text Aphasia Battery (TAB) — paper by Roll et al. (2026)"
description: "The authors worked with speech-language pathologists to build the Text Aphasia Battery (TAB), a text-only adaptation of the… 19 references."
authors: ["Nathan Roll", "Jill Kries", "Flora Jin", "Catherine Wang", "Ann Marie Finley", "Meghan Sumner", "Cory Shain", "Laura Gwilliams"]
year: 2026
venue: "Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2026)"
source: "https://aclanthology.org/2026.clpsych-1.27/"
doi: "10.18653/v1/2026.clpsych-1.27"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/roll-2026-text-aphasia-battery-tab-clinically/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/roll-2026-text-aphasia-battery-tab-clinically/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/53/) / The Text Aphasia Battery (TAB): A Clinically-Grounded Benchmark for Aphasia-Like Deficits in Language Models

# The Text Aphasia Battery (TAB): A Clinically-Grounded Benchmark for Aphasia-Like Deficits in Language Models

**Authors:** [Nathan Roll](https://md.research.elamur.ai/free/en/authors/nathan-roll/), Jill Kries, Flora Jin, Catherine Wang, Ann Marie Finley, Meghan Sumner, [Cory Shain](https://md.research.elamur.ai/free/en/authors/cory-shain/) and Laura Gwilliams · **Year:** 2026 · **Venue:** [Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2026)](https://md.research.elamur.ai/free/en/venues/clpsych/) ([Workshop on Computational Linguistics and Clinical Psychology 2026](https://md.research.elamur.ai/free/en/venues/clpsych/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.clpsych-1.27/) · [PDF](https://aclanthology.org/2026.clpsych-1.27.pdf) · [DOI 10.18653/v1/2026.clpsych-1.27](https://doi.org/10.18653/v1/2026.clpsych-1.27) · [OpenAlex](https://openalex.org/W4416773944)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Clinical aphasia tests assume a human patient who can see pictures, hear speech, gesture and reflect on their own performance. Text-only LLMs can do none of this, so these tests cannot be applied to them directly. The authors worked with speech-language pathologists to build the Text Aphasia Battery (TAB), a text-only adaptation of the Quick Aphasia Battery (QAB). It keeps the language dimensions that remain visible in transcripts: word retrieval, syntax, discourse coherence and faithful repetition. Its four subtests draw their scoring criteria from QAB and the APROCSA rating system.

To make large-scale use practical, the team tested Gemini 2.5 Flash as an automatic rater against expert annotators. When features are weighted by how often they occur, the model's agreement with the expert consensus is roughly as high as the agreement between two human experts. Baseline runs on small open models show that TAB separates models by task compliance and output quality. The authors present these runs as calibration, not as evidence of "aphasia" in models. The benchmark is meant as the measurement tool for companion work on artificially lesioned models.

## Key points
- **Four subtests:**
  - Connected Text: open prompts scored on 19 APROCSA-based features.
  - Word Comprehension: forced choice among six options.
  - Sentence Comprehension: yes/no items covering passives, negation and conditionals.
  - Repetition: verbatim reproduction of items that grow longer and more complex.
- **Item counts:** each subtest administers five items. The full prompt pool contains about 104 prompts.
- **Rater validation:** prevalence-weighted Cohen's κ was 0.255 for Gemini 2.5 Flash vs. consensus and 0.286 for human vs. human. The authors caution that unweighted averaging can mislead when some features are rare.
- **Baseline symptom burden** (mean symptoms per response):
  - Llama-3.2-1B: 3.21
  - OLMo-2-1B: 1.98
  - OLMo-2-1B-Instruct: 0.83
  - Gemma-3-1B-it: 0.35
  - Llama-3.2-1B-Instruct: 0.15
- **Release and scope:** TAB is released under the MIT License. It is English-only, not intended for clinical diagnosis, and uses AphasiaBank as a human reference resource.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,053 citations
- [University of California San Diego](https://md.research.elamur.ai/free/en/universities/university-of-california-san-diego/) — University, San Diego, United States · 89 works · 54,032 citations
- [Temple University](https://md.research.elamur.ai/free/en/universities/temple-university/) — University, Philadelphia, United States · 11 works · 24,319 citations

Also on the byline: [San Diego State University](https://ror.org/0264fdx42).

📚 Shelves (8): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,649 works · [💭 Developmental and Educational Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/developmental-and-educational-psychology/) · 523 works · [🧠 Neuroscience](https://md.research.elamur.ai/free/en/shelves/neuroscience/) · 186 works · [🧠 Cognitive Neuroscience](https://md.research.elamur.ai/free/en/shelves/neuroscience/cognitive-neuroscience/) · 185 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works

## Cite

Cite as: Nathan Roll, Jill Kries, Flora Jin, Catherine Wang, Ann Marie Finley, Meghan Sumner, Cory Shain and Laura Gwilliams. 2026. The Text Aphasia Battery (TAB): A Clinically-Grounded Benchmark for Aphasia-Like Deficits in Language Models. Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{roll2026text,
  title = {{The Text Aphasia Battery (TAB): A Clinically-Grounded Benchmark for Aphasia-Like Deficits in Language Models}},
  author = {Nathan Roll, Jill Kries, Flora Jin, Catherine Wang, Ann Marie Finley, Meghan Sumner, Cory Shain and Laura Gwilliams},
  booktitle = {Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.clpsych-1.27},
  url = {https://doi.org/10.18653/v1/2026.clpsych-1.27}
}
```

- [← previous The Texas German Dialect Project Corpus as a Diachronic Resource for Investigating Language Contact](https://md.research.elamur.ai/free/en/papers/schmidt-2026-texas-german-dialect-project-corpus/)
- [next → The Validity of Coreference-based Evaluations of Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/porada-2026-validity-coreference-based-evaluations-natural/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/53/)

## References

19 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. 2023. Pythia: A suite for analyzing large language models across… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Gregory+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+Usvsn+Sai+Prashanth%2C+Edward+Raff%2C+Aviya+Skowron%2C+Linta)
2. Carolyn Bruce and Anne Edmundson. 2010. Letting the CAT out of the bag: A review of the Comprehensive Aphasia Test. commentary on howard, swinburn, and porter, “putting the CAT out: What the Comprehensive Aphasia Test has to offer”. Aphasiology, 24(1):79–93. [find](https://scholar.google.com/scholar?q=Carolyn+Bruce+and+Anne+Edmundson%2E+2010%2E+Letting+the+CAT+out+of+the+bag%3A+A+review+of+the+Comprehensive+Aphasia+Test%2E+commentary+on+howard%2C+swinburn%2C+and+porter%2C+%E2%80%9Cputting+the+CAT+out%3A+What+the+Comprehen)
3. Marianne Casilio, Kindle Rising, Pélagie M. Beeson, Kate Bunton, and Stephen M. Wilson. 2019. Auditory-perceptual rating of connected speech in aphasia. American Journal of Speech-Language Pathology, 28(2):550–568. [find](https://scholar.google.com/scholar?q=Marianne+Casilio%2C+Kindle+Rising%2C+P%C3%A9lagie+M%2E+Beeson%2C+Kate+Bunton%2C+and+Stephen+M%2E+Wilson%2E+2019%2E+Auditory%2Dperceptual+rating+of+connected+speech+in+aphasia%2E+American+Journal+of+Speech%2DLanguage+Pathology%2C+)
4. Heather M. Clark, Rene L. Utianski, Joseph R. Duffy, Edythe A. Strand, Hugo Botha, Keith A. Josephs, and Jennifer L. Whitwell. 2020. Western Aphasia Battery–Revised profiles in primary progressive aphasia and primary progressive apraxia of speech. American Journal of Speech-Language Pathology, 29(1S… [find](https://scholar.google.com/scholar?q=Heather+M%2E+Clark%2C+Rene+L%2E+Utianski%2C+Joseph+R%2E+Duffy%2C+Edythe+A%2E+Strand%2C+Hugo+Botha%2C+Keith+A%2E+Josephs%2C+and+Jennifer+L%2E+Whitwell%2E+2020%2E+Western+Aphasia+Battery%E2%80%93Revised+profiles+in+primary+progressive+aph)
5. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4):213–220. [find](https://scholar.google.com/scholar?q=Jacob+Cohen%2E+1968%2E+Weighted+kappa%3A+Nominal+scale+agreement+provision+for+scaled+disagreement+or+partial+credit%2E+Psychological+Bulletin%2C+70%284%29%3A213%E2%80%93220%2E)
6. Yan Cong, Arianna N. LaCroix, and Jiyeon Lee. 2024. Clinical efficacy of pre-trained large language models through the lens of aphasia. Scientific Reports, 14(1):15573. [find](https://scholar.google.com/scholar?q=Yan+Cong%2C+Arianna+N%2E+LaCroix%2C+and+Jiyeon+Lee%2E+2024%2E+Clinical+efficacy+of+pre%2Dtrained+large+language+models+through+the+lens+of+aphasia%2E+Scientific+Reports%2C+14%281%29%3A15573%2E)
7. Margaret Forbes, Davida Fromm, and Brian MacWhinney. 2012. Aphasiabank: A resource for clinicians. Seminars in Speech and Language, 33(3):217–222. [find](https://scholar.google.com/scholar?q=Margaret+Forbes%2C+Davida+Fromm%2C+and+Brian+MacWhinney%2E+2012%2E+Aphasiabank%3A+A+resource+for+clinicians%2E+Seminars+in+Speech+and+Language%2C+33%283%29%3A217%E2%80%93222%2E)
8. Ajay D. Halai, Blanca De Dios Perez, James D. Stefaniak, and Matthew A. Lambon Ralph. 2022. Efficient and effective assessment of deficits and their neural bases in stroke aphasia. Cortex, 155:333–346. [find](https://scholar.google.com/scholar?q=Ajay+D%2E+Halai%2C+Blanca+De+Dios+Perez%2C+James+D%2E+Stefaniak%2C+and+Matthew+A%2E+Lambon+Ralph%2E+2022%2E+Efficient+and+effective+assessment+of+deficits+and+their+neural+bases+in+stroke+aphasia%2E+Cortex%2C+155%3A333%E2%80%93346)
9. Judge Dependence and Robustness Automated evaluation currently relies on a single judge model and prompt. Future work will evaluate robustness across multiple judge models and prompts and include human adjudication of disagreements to bound judge-specific bias. [find](https://scholar.google.com/scholar?q=Judge+Dependence+and+Robustness+Automated+evaluation+currently+relies+on+a+single+judge+model+and+prompt%2E+Future+work+will+evaluate+robustness+across+multiple+judge+models+and+prompts+and+include+huma)
10. Andrew Kertesz. 2022. The Western Aphasia Battery: A systematic review of research and clinical applications. Aphasiology, 36(1):21–50. [find](https://scholar.google.com/scholar?q=Andrew+Kertesz%2E+2022%2E+The+Western+Aphasia+Battery%3A+A+systematic+review+of+research+and+clinical+applications%2E+Aphasiology%2C+36%281%29%3A21%E2%80%9350%2E)
11. Psychometrics and Item Design Future work will increase subtest difficulty to mitigate ceiling effects (e.g., harder foils, broader syntactic phenomena, balanced Yes/No) and perform item-response analysis to estimate item difficulty and discrimination. We will also analyze repetition errors under a… [find](https://scholar.google.com/scholar?q=Psychometrics+and+Item+Design+Future+work+will+increase+subtest+difficulty+to+mitigate+ceiling+effects+%28e%2Eg%2E%2C+harder+foils%2C+broader+syntactic+phenomena%2C+balanced+Yes%2FNo%29+and+perform+item%2Dresponse+anal)
12. Brian MacWhinney, Davida Fromm, Margaret Forbes, and Audrey Holland. 2011. Aphasiabank: Methods for studying discourse. Aphasiology, 25(11):1286– 1307. [find](https://scholar.google.com/scholar?q=Brian+MacWhinney%2C+Davida+Fromm%2C+Margaret+Forbes%2C+and+Audrey+Holland%2E+2011%2E+Aphasiabank%3A+Methods+for+studying+discourse%2E+Aphasiology%2C+25%2811%29%3A1286%E2%80%93+1307%2E)
13. Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2024. Dissociating language and thought in large language models. Trends in Cognitive Sciences, 28(6):517–540. [find](https://scholar.google.com/scholar?q=Kyle+Mahowald%2C+Anna+A%2E+Ivanova%2C+Idan+A%2E+Blank%2C+Nancy+Kanwisher%2C+Joshua+B%2E+Tenenbaum%2C+and+Evelina+Fedorenko%2E+2024%2E+Dissociating+language+and+thought+in+large+language+models%2E+Trends+in+Cognitive+Scienc)
14. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, OpenAI. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+Technical+report%2C+OpenAI%2E)
15. Scoring: For each response, identify the presence (1) or absence (0) of APROCSA-based aphasic features (see Section B.5). B.2 [find](https://scholar.google.com/scholar?q=Scoring%3A+For+each+response%2C+identify+the+presence+%281%29+or+absence+%280%29+of+APROCSA%2Dbased+aphasic+features+%28see+Section+B%2E5%29%2E+B%2E2)
16. Nathan Roll, Jill Kries, Laura Gwilliams, and Cory Shain. 2026. Artificial aphasias in lesioned language models. Preprint, arXiv:2605.16222. — [arXiv:2605.16222](https://arxiv.org/abs/2605.16222) [find](https://scholar.google.com/scholar?q=Nathan+Roll%2C+Jill+Kries%2C+Laura+Gwilliams%2C+and+Cory+Shain%2E+2026%2E+Artificial+aphasias+in+lesioned+language+models%2E+Preprint%2C+arXiv%3A2605%2E16222%2E+%E2%80%94+%5BarXiv%3A2605%2E16222%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2605%2E16222%29)
17. Katherine Salter, Jeffrey Jutai, Norine Foley, Chelsea Hellings, and Robert Teasell. 2006. Identification of aphasia post stroke: A review of screening assessment tools. Brain Injury, 20(6):559–568. [find](https://scholar.google.com/scholar?q=Katherine+Salter%2C+Jeffrey+Jutai%2C+Norine+Foley%2C+Chelsea+Hellings%2C+and+Robert+Teasell%2E+2006%2E+Identification+of+aphasia+post+stroke%3A+A+review+of+screening+assessment+tools%2E+Brain+Injury%2C+20%286%29%3A559%E2%80%93568%2E)
18. Subtest 2: Word Comprehension Objective: Evaluates lexical-semantic processing and selection among competing meanings. Instructions: Elicit responses verbatim. Provide the options alongside each question. Require a one-token answer corresponding exactly to one of the listed options; do not include e… [find](https://scholar.google.com/scholar?q=Subtest+2%3A+Word+Comprehension+Objective%3A+Evaluates+lexical%2Dsemantic+processing+and+selection+among+competing+meanings%2E+Instructions%3A+Elicit+responses+verbatim%2E+Provide+the+options+alongside+each+quest)
19. Luise Springer and Stefanie Mantey. 2010. [The Comprehensive Aphasia Test: A review](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). commentary on howard, swinburn, and porter, “putting the CAT out: What the Comprehensive Aphasia Test has to offer”. Aphasiology, 24(1):75–78. 1. “Which one is an animal with a mane? (lion, drum, violin, giraffe, boo…

*Updated 1 November 2026*