---
title: "Towards Consistent Detection of Cognitive — Sharma 2026"
description: "Detecting cognitive distortions (CDs) in text is difficult because the task is subjective."
authors: ["Neha Sharma", "Navneet Agarwal", "Kairit Sirts"]
year: 2026
venue: "Proceedings of the Fifteenth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2026.lrec-1.851/"
doi: "10.63317/4joqxtgdgxq2"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/sharma-2026-consistent-detection-cognitive-distortions-llm/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/sharma-2026-consistent-detection-cognitive-distortions-llm/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2026](https://md.research.elamur.ai/free/papers/2026/page/5/) / Towards Consistent Detection of Cognitive Distortions: LLM-Based Annotation and Dataset-Agnostic Evaluation

# Towards Consistent Detection of Cognitive Distortions: LLM-Based Annotation and Dataset-Agnostic Evaluation

**Authors:** Neha Sharma, Navneet Agarwal, [Kairit Sirts](https://md.research.elamur.ai/free/authors/kairit-sirts/) · **Year:** 2026 · **Venue:** Proceedings of the Fifteenth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.lrec-1.851/) · [PDF](https://aclanthology.org/2026.lrec-1.851.pdf) · [DOI 10.63317/4joqxtgdgxq2](https://doi.org/10.63317/4joqxtgdgxq2) · [OpenAlex](https://openalex.org/W4416437295)

**Citations:** 1 citations (openalex-snapshot · 2026-10-14) · 1 in our library, of the 140673 papers whose bibliographies we read

## Summary
Detecting cognitive distortions (CDs) in text is difficult because the task is subjective. Even expert annotators agree poorly, so existing training labels are noisy, and reported classifier scores remain low. The authors re-annotate the public Therapist Q&A dataset with GPT models, running several independent annotation passes per text. They keep only labels that recur across nearly all runs and treat this stability as evidence of reliability. They also propose a dataset-agnostic evaluation that treats Cohen's kappa as an effect size, scoring models relative to a chance baseline instead of relying on raw F1. This allows fair comparison across datasets and studies with different label distributions. GPT-4 produced highly consistent labels across runs. Classifiers trained on the LLM labels beat those trained on the original human labels under the new metric. A check by domain experts, however, showed no clear preference between the two label sources. The work offers a scalable route to internally consistent training data for subjective NLP tasks. It also shows that texts describing events rather than thoughts may be inherently unsuitable for CD detection.

## Key points
- The data is the Therapist Q&A dataset (Kaggle): 2530 user input–response pairs labeled by Shreevastava and Foltz (2021). It uses 10 CD categories plus "No Distortion", and the original inter-annotator agreement was only 33.7% (joint probability).
- Annotation used GPT-4 and GPT-4o via Microsoft Azure at temperatures 0.5 and 0.7. Two prompt types were tested: a Multi-Label Prompt (MLP) and a Ranked-Label Prompt (RLP) that selects one dominant CD plus an optional secondary one.
- GPT-4 reached an inter-run agreement of Fleiss' kappa = 0.78. Earlier CD classifiers typically scored a weighted F1 of only about 0.2–0.4.
- The proposed kappa-based effect size (κF1) is presented as transferable to any field with unstable benchmarks.
- Limitations: only GPT-family models were tested, the experts disagreed among themselves, and the provenance of the dataset is unverifiable. The code and the LLM-generated labels are released on GitHub.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sharma2026consistent,
  title = {{Towards Consistent Detection of Cognitive Distortions: LLM-Based Annotation and Dataset-Agnostic Evaluation}},
  author = {Sharma, Neha and Agarwal, Navneet and Sirts, Kairit},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4joqxtgdgxq2},
  url = {https://doi.org/10.63317/4joqxtgdgxq2}
}
```

- [← previous Towards a Universal Dependency Corpus for Old Saxon (Old Low German)](https://md.research.elamur.ai/free/papers/chiarcos-2026-universal-dependency-corpus-old-saxon/)
- [next → Towards Dynamic Attention Masking for Simultaneous Speech Translation](https://md.research.elamur.ai/free/papers/pong-2026-dynamic-attention-masking-simultaneous-speech/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/papers/2026/page/5/)

## References

29 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mohammed Aldeen, Joshua Luo, Ashley Lian, Venus Zheng, Allen Hong, Preethika Yetukuri, and Long Cheng. 2023. Chatgpt vs. human annotators: A comprehensive analysis of chatgpt for text annotation. In 2023 International Conference on Machine Learning and Applications (ICMLA), pages 602–609. [find](https://scholar.google.com/scholar?q=Mohammed+Aldeen%2C+Joshua+Luo%2C+Ashley+Lian%2C+Venus+Zheng%2C+Allen+Hong%2C+Preethika+Yetukuri%2C+and+Long+Cheng%2E+2023%2E+Chatgpt+vs%2E+human+annotators%3A+A+comprehensive+analysis+of+chatgpt+for+text+annotation%2E+In+2)
2. Jelly P. Aureus, Ma. Regina Justina E. Estuar, Dorothy C. Mapua, Roland P. Abao, and Anna Angeline M. Cataluña. 2021. Determining linguistic markers in cognitive distortions from covid-19 pandemic-related reddit texts. In 2021 1st International Conference in Information and Computing Research (iCORE… [find](https://scholar.google.com/scholar?q=Jelly+P%2E+Aureus%2C+Ma%2E+Regina+Justina+E%2E+Estuar%2C+Dorothy+C%2E+Mapua%2C+Roland+P%2E+Abao%2C+and+Anna+Angeline+M%2E+Catalu%C3%B1a%2E+2021%2E+Determining+linguistic+markers+in+cognitive+distortions+from+covid%2D19+pandemic%2Drel)
3. Aaron T Beck. 1963. Thinking and depression: I. idiosyncratic content and cognitive distortions. Archives of general psychiatry, 9(4):324–333. [find](https://scholar.google.com/scholar?q=Aaron+T+Beck%2E+1963%2E+Thinking+and+depression%3A+I%2E+idiosyncratic+content+and+cognitive+distortions%2E+Archives+of+general+psychiatry%2C+9%284%29%3A324%E2%80%93333%2E)
4. Aaron T Beck and Brad A Alford. 2009. Depression: Causes and treatment. University of Pennsylvania Press. [find](https://scholar.google.com/scholar?q=Aaron+T+Beck+and+Brad+A+Alford%2E+2009%2E+Depression%3A+Causes+and+treatment%2E+University+of+Pennsylvania+Press%2E)
5. David D Burns. 1989. The feeling good handbook: Using the new mood therapy in everyday life. William Morrow & Co. [find](https://scholar.google.com/scholar?q=David+D+Burns%2E+1989%2E+The+feeling+good+handbook%3A+Using+the+new+mood+therapy+in+everyday+life%2E+William+Morrow+%26+Co%2E)
6. David D Burns and MD Feeling Good. 1980. The new mood therapy. 6 [https://github.com/nehasharma666/llm-cognitive-distortion-detection](https://github.com/nehasharma666/llm-cognitive-distortion-detection) [↗](https://github.com/nehasharma666/llm-cognitive-distortion-detection)
7. Zhiyu Chen, Yujie Lu, and William Wang. 2023. Empowering psychotherapy with large language models: Cognitive distortion detection through diagnosis of thought prompting. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4295–4304, Singapore. Association for Computationa… [find](https://scholar.google.com/scholar?q=Zhiyu+Chen%2C+Yujie+Lu%2C+and+William+Wang%2E+2023%2E+Empowering+psychotherapy+with+large+language+models%3A+Cognitive+distortion+detection+through+diagnosis+of+thought+prompting%2E+In+Findings+of+the+Association)
8. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. [find](https://scholar.google.com/scholar?q=Jacob+Cohen%2E+1960%2E+A+coefficient+of+agreement+for+nominal+scales%2E+Educational+and+psychological+measurement%2C+20%281%29%3A37%E2%80%9346%2E)
9. Xiruo Ding, Kevin Lybarger, Justin Tauscher, and Trevor Cohen. 2022. Improving classification of infrequent cognitive distortions: Domain-specific model vs. data augmentation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human… [find](https://scholar.google.com/scholar?q=Xiruo+Ding%2C+Kevin+Lybarger%2C+Justin+Tauscher%2C+and+Trevor+Cohen%2E+2022%2E+Improving+classification+of+infrequent+cognitive+distortions%3A+Domain%2Dspecific+model+vs%2E+data+augmentation%2E+In+Proceedings+of+the+20)
10. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. [find](https://scholar.google.com/scholar?q=Joseph+L+Fleiss%2E+1971%2E+Measuring+nominal+scale+agreement+among+many+raters%2E+Psychological+bulletin%2C+76%285%29%3A378%2E)
11. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30):e2305016120. [find](https://scholar.google.com/scholar?q=Fabrizio+Gilardi%2C+Meysam+Alizadeh%2C+and+Ma%C3%ABl+Kubli%2E+2023%2E+Chatgpt+outperforms+crowd+workers+for+text%2Dannotation+tasks%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+120%2830%29%3Ae2305016120%2E) [our card](https://md.research.elamur.ai/free/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/)
12. Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024. Annollm: Making large language models to be better crowdsourced annotators. [find](https://scholar.google.com/scholar?q=Xingwei+He%2C+Zhenghao+Lin%2C+Yeyun+Gong%2C+A%2DLong+Jin%2C+Hang+Zhang%2C+Chen+Lin%2C+Jian+Jiao%2C+Siu+Ming+Yiu%2C+Nan+Duan%2C+and+Weizhu+Chen%2E+2024%2E+Annollm%3A+Making+large+language+models+to+be+better+crowdsourced+annota) [our card](https://md.research.elamur.ai/free/papers/he-2023-annollm-making-large-language-models/)
13. Peter Henderson and Emma Brunskill. 2018. Distilling information from a flood: A possibility for the use of meta-analysis and systematic review in machine learning research. CoRR, abs/1812.01074. — [arXiv:1812.01074](https://arxiv.org/abs/1812.01074) [find](https://scholar.google.com/scholar?q=Peter+Henderson+and+Emma+Brunskill%2E+2018%2E+Distilling+information+from+a+flood%3A+A+possibility+for+the+use+of+meta%2Danalysis+and+systematic+review+in+machine+learning+research%2E+CoRR%2C+abs%2F1812%2E01074%2E+%E2%80%94+%5Ba)
14. Michael Heseltine and Bernhard Clemm von Hohenberg. 2024. Large language models as a substitute for human experts in annotating political text. Research & Politics, 11(1):20531680241236239. [find](https://scholar.google.com/scholar?q=Michael+Heseltine+and+Bernhard+Clemm+von+Hohenberg%2E+2024%2E+Large+language+models+as+a+substitute+for+human+experts+in+annotating+political+text%2E+Research+%26+Politics%2C+11%281%29%3A20531680241236239%2E)
15. Shaoxiong Ji, Tianlin Zhang, Luna Ansari, Jie Fu, Prayag Tiwari, and Erik Cambria. 2022. Mental-BERT: Publicly Available Pretrained Language Models for Mental Healthcare. [find](https://scholar.google.com/scholar?q=Shaoxiong+Ji%2C+Tianlin+Zhang%2C+Luna+Ansari%2C+Jie+Fu%2C+Prayag+Tiwari%2C+and+Erik+Cambria%2E+2022%2E+Mental%2DBERT%3A+Publicly+Available+Pretrained+Language+Models+for+Mental+Healthcare%2E)
16. Jutta Joormann and Colin H Stanton. 2016. Examining emotion regulation in depression: A review and future directions. Behaviour research and therapy, 86:35–49. [find](https://scholar.google.com/scholar?q=Jutta+Joormann+and+Colin+H+Stanton%2E+2016%2E+Examining+emotion+regulation+in+depression%3A+A+review+and+future+directions%2E+Behaviour+research+and+therapy%2C+86%3A35%E2%80%9349%2E)
17. Ken Kelley and Kristopher J Preacher. 2012. On effect size. Psychological methods, 17(2):137. [find](https://scholar.google.com/scholar?q=Ken+Kelley+and+Kristopher+J+Preacher%2E+2012%2E+On+effect+size%2E+Psychological+methods%2C+17%282%29%3A137%2E)
18. Jiyi Li. 2024. A comparative study on annotation quality of crowdsourcing and llm via label aggregation. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6525–6529. [find](https://scholar.google.com/scholar?q=Jiyi+Li%2E+2024%2E+A+comparative+study+on+annotation+quality+of+crowdsourcing+and+llm+via+label+aggregation%2E+In+ICASSP+2024+%2D+2024+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+)
19. Sehee Lim, Yejin Kim, Chi-Hyun Choi, Jy-yong Sohn, and Byung-Hoon Kim. 2024. Erd: a framework for improving llm reasoning for cognitive distortion classification. [find](https://scholar.google.com/scholar?q=Sehee+Lim%2C+Yejin+Kim%2C+Chi%2DHyun+Choi%2C+Jy%2Dyong+Sohn%2C+and+Byung%2DHoon+Kim%2E+2024%2E+Erd%3A+a+framework+for+improving+llm+reasoning+for+cognitive+distortion+classification%2E)
20. Kevin Lybarger, Justin Tauscher, Xiruo Ding, Dror Ben-zeev, and Trevor Cohen. 2022. Identifying distorted thinking in patient-therapist text message exchanges by leveraging dynamic multi-turn context. In Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology, pages 1… [find](https://scholar.google.com/scholar?q=Kevin+Lybarger%2C+Justin+Tauscher%2C+Xiruo+Ding%2C+Dror+Ben%2Dzeev%2C+and+Trevor+Cohen%2E+2022%2E+Identifying+distorted+thinking+in+patient%2Dtherapist+text+message+exchanges+by+leveraging+dynamic+multi%2Dturn+context%2E)
21. Nawal Ouhmad, Romain Deperrois, Wissam El Hage, and Nicolas Combalbert. 2024. Cognitive distortions, anxiety, and depression in individuals suffering from ptsd. International Journal of Mental Health, 53(4):336–352. [find](https://scholar.google.com/scholar?q=Nawal+Ouhmad%2C+Romain+Deperrois%2C+Wissam+El+Hage%2C+and+Nicolas+Combalbert%2E+2024%2E+Cognitive+distortions%2C+anxiety%2C+and+depression+in+individuals+suffering+from+ptsd%2E+International+Journal+of+Mental+Health%2C)
22. Lina Rojas-Barahona, Bo-Hsiang Tseng, Yinpei Dai, Clare Mansfield, Osman Ramadan, Stefan Ultes, Michael Crawford, and Milica Gasic. 2018. Deep learning for language understanding of mental health concepts derived from cognitive behavioural therapy. [find](https://scholar.google.com/scholar?q=Lina+Rojas%2DBarahona%2C+Bo%2DHsiang+Tseng%2C+Yinpei+Dai%2C+Clare+Mansfield%2C+Osman+Ramadan%2C+Stefan+Ultes%2C+Michael+Crawford%2C+and+Milica+Gasic%2E+2018%2E+Deep+learning+for+language+understanding+of+mental+health+conc)
23. Benjamin Shickel, Scott Siegel, Martin Heesacker, Sherry Benton, and Parisa Rashidi. 2020. Automatic detection and classification of cognitive distortions in mental health text. In 2020 IEEE 20th International Conference on Bioinformatics and Bioengineering (BIBE), pages 275–280. [find](https://scholar.google.com/scholar?q=Benjamin+Shickel%2C+Scott+Siegel%2C+Martin+Heesacker%2C+Sherry+Benton%2C+and+Parisa+Rashidi%2E+2020%2E+Automatic+detection+and+classification+of+cognitive+distortions+in+mental+health+text%2E+In+2020+IEEE+20th+Inte)
24. Sagarika Shreevastava and Peter Foltz. 2021. Detecting cognitive distortions from patient-therapist interactions. In Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access, pages 151–158, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sagarika+Shreevastava+and+Peter+Foltz%2E+2021%2E+Detecting+cognitive+distortions+from+patient%2Dtherapist+interactions%2E+In+Proceedings+of+the+Seventh+Workshop+on+Computational+Linguistics+and+Clinical+Psych)
25. T. Simms, C. Ramstedt, M. Rich, M. Richards, T. Martinez, and C. Giraud-Carrier. 2017. Detecting cognitive distortions through machine learning text analytics. In 2017 IEEE International Conference on Healthcare Informatics (ICHI), pages 508–512. [find](https://scholar.google.com/scholar?q=T%2E+Simms%2C+C%2E+Ramstedt%2C+M%2E+Rich%2C+M%2E+Richards%2C+T%2E+Martinez%2C+and+C%2E+Giraud%2DCarrier%2E+2017%2E+Detecting+cognitive+distortions+through+machine+learning+text+analytics%2E+In+2017+IEEE+International+Conference+on)
26. Justin S Tauscher, Kevin Lybarger, Xiruo Ding, Ayesha Chander, William J Hudenko, Trevor Cohen, and Dror Ben-Zeev. 2023. Automated detection of cognitive distortions in text exchanges between clinicians and people with serious mental illness. Psychiatric services, 74(4):407–410. [find](https://scholar.google.com/scholar?q=Justin+S+Tauscher%2C+Kevin+Lybarger%2C+Xiruo+Ding%2C+Ayesha+Chander%2C+William+J+Hudenko%2C+Trevor+Cohen%2C+and+Dror+Ben%2DZeev%2E+2023%2E+Automated+detection+of+cognitive+distortions+in+text+exchanges+between+clinicia)
27. Bichen Wang, Pengfei Deng, Yanyan Zhao, and Bing Qin. 2023. C2D2 dataset: A resource for the cognitive distortion analysis and its impact on mental health. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10149–10160, Singapore. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Bichen+Wang%2C+Pengfei+Deng%2C+Yanyan+Zhao%2C+and+Bing+Qin%2E+2023%2E+C2D2+dataset%3A+A+resource+for+the+cognitive+distortion+analysis+and+its+impact+on+mental+health%2E+In+Findings+of+the+Association+for+Computati)
28. Gülin Yazici-Çelebi and Feridun Kaya. 2022. Interpersonal cognitive distortions and anxiety: The mediating role of emotional intelligence. International Journal of Psychology and Educational Studies, 9(3):741–753. [find](https://scholar.google.com/scholar?q=G%C3%BClin+Yazici%2D%C3%87elebi+and+Feridun+Kaya%2E+2022%2E+Interpersonal+cognitive+distortions+and+anxiety%3A+The+mediating+role+of+emotional+intelligence%2E+International+Journal+of+Psychology+and+Educational+Studies%2C+)
29. Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. 2023. Llmaaa: Making large language models as active annotators. [find](https://scholar.google.com/scholar?q=Ruoyu+Zhang%2C+Yanzeng+Li%2C+Yongliang+Ma%2C+Ming+Zhou%2C+and+Lei+Zou%2E+2023%2E+Llmaaa%3A+Making+large+language+models+as+active+annotators%2E)