---
title: "Large-scale Analysis of Counseling Conversations: An"
description: "Evidence on what makes a counseling conversation work has mostly come from small… 0.638 AUC. 40 references, cited by 67 papers here."
authors: ["Tim Althoff", "Kevin Clark", "Jure Leskovec"]
year: 2016
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/Q16-1033/"
doi: "10.1162/tacl_a_00111"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/althoff-2016-large-scale-analysis-counseling-conversations/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/althoff-2016-large-scale-analysis-counseling-conversations/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/) / Large-scale Analysis of Counseling Conversations: An Application of Natural Language Processing to Mental Health

# Large-scale Analysis of Counseling Conversations: An Application of Natural Language Processing to Mental Health

**Authors:** [Tim Althoff](https://md.research.elamur.ai/free/en/authors/tim-althoff/), [Kevin Clark](https://md.research.elamur.ai/free/en/authors/kevin-clark/) and [Jure Leskovec](https://md.research.elamur.ai/free/en/authors/jure-leskovec/) · **Year:** 2016 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2016](https://md.research.elamur.ai/free/en/venues/tacl/2016/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/Q16-1033/) · [PDF](https://aclanthology.org/Q16-1033.pdf) · [DOI 10.1162/tacl\_a\_00111](https://doi.org/10.1162/tacl_a_00111) · [OpenAlex](https://openalex.org/W2484061479)

**Citations:** 323 citations (OpenAlex, 2026-10-14) · [67](https://md.research.elamur.ai/free/en/papers/althoff-2016-large-scale-analysis-counseling-conversations/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Evidence on what makes a counseling conversation work has mostly come from small, qualitative studies. Althoff, Clark and Leskovec address this with a large quantitative analysis of an SMS-based crisis counseling service. After each conversation, the texter reported whether they felt better, and this answer served as the outcome label. The authors compare the counselors with the highest success rates against those with the lowest. They use several computational discourse techniques: comparisons of language models, clustering of counselor replies to similar ambiguous situations, an unsupervised HMM-style model of ordered conversation stages, and word-frequency measures drawn from psycholinguistics.

Stronger counselors behave differently in several ways:
- They adjust to how the dialogue is going.
- They write longer, clarifying replies and check their understanding.
- They avoid templated or copied wording.
- They identify the core problem sooner and move faster to joint problem-solving.
- They help texters shift toward the future, toward other people and toward positive sentiment.

Features built on these strategies predict the outcome of individual conversations. The work points toward better counselor training and real-time support tools, and the dataset is publicly available.

## Key points
- The corpus covers November 2013 to November 2014 and contains 80,885 conversations, about 3.2 million messages and 408 counselors. Of these conversations, 15,555 (19.2%) have a follow-up label, coded as "better" versus "same/worse".
- The counselor comparison draws on 113 counselors with enough long labeled conversations. The top 40 had success rates of 66.3–85.5%, and the bottom 40 had rates of 42.1–58.6%.
- The prediction model is a logistic regression trained on balanced data built around 3,619 negative conversations, evaluated with 10-fold cross-validation and scored by AUC.
- With 80% of a conversation visible, the results were:
  - n-grams alone: 0.638 AUC
  - full model using only counselor features: 0.652 AUC
  - full model plus texter n-grams: 0.708 AUC
- The model can separate outcomes after seeing only the first 20% of a conversation. Access details for the dataset are at snap.stanford.edu/counseling.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 310 works · 284,480 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 798 works · [💭 Social Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/social-psychology/) · 129 works · [💭 Applied Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/applied-psychology/) · 6 works

## Cite

Cite as: Tim Althoff, Kevin Clark and Jure Leskovec. 2016. Large-scale Analysis of Counseling Conversations: An Application of Natural Language Processing to Mental Health. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{althoff2016large,
  title = {{Large-scale Analysis of Counseling Conversations: An Application of Natural Language Processing to Mental Health}},
  author = {Tim Althoff, Kevin Clark and Jure Leskovec},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.1162/tacl\_a\_00111},
  url = {https://doi.org/10.1162/tacl_a_00111}
}
```

- [← previous CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://md.research.elamur.ai/free/en/papers/johnson-2016-clevr-diagnostic-dataset-compositional-language/)
- [next → Multi30K: Multilingual English-German Image Descriptions](https://md.research.elamur.ai/free/en/papers/elliott-2016-multi30k-multilingual-english-german-image/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/)

## References

40 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tim Althoff and Jure Leskovec. 2015. Donor retention in online crowdfunding communities: A case study of DonorsChoose.org. In WWW. [find](https://scholar.google.com/scholar?q=Tim+Althoff+and+Jure+Leskovec%2E+2015%2E+Donor+retention+in+online+crowdfunding+communities%3A+A+case+study+of+DonorsChoose%2Eorg%2E+In+WWW%2E)
2. Tim Althoff, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2014. How to ask for a favor: A case study on the success of altruistic requests. In ICWSM. [find](https://scholar.google.com/scholar?q=Tim+Althoff%2C+Cristian+Danescu%2DNiculescu%2DMizil%2C+and+Dan+Jurafsky%2E+2014%2E+How+to+ask+for+a+favor%3A+A+case+study+on+the+success+of+altruistic+requests%2E+In+ICWSM%2E)
3. Vikas Ganjigunte Ashok, Song Feng, and Yejin Choi. 2013. Success with style: Using writing style to predict the success of novels. In EMNLP. [find](https://scholar.google.com/scholar?q=Vikas+Ganjigunte+Ashok%2C+Song+Feng%2C+and+Yejin+Choi%2E+2013%2E+Success+with+style%3A+Using+writing+style+to+predict+the+success+of+novels%2E+In+EMNLP%2E)
4. Regina Barzilay and Lillian Lee. 2004. [Catching the drift: Probabilistic content models, with applications to generation and summarization](https://md.research.elamur.ai/free/en/papers/barzilay-2004-catching-drift-probabilistic-content-models/). In HLT-NAACL.
5. Aaron T. Beck. 1967. Depression: Clinical, experimental, and theoretical aspects. University of Pennsylvania Press. [find](https://scholar.google.com/scholar?q=Aaron+T%2E+Beck%2E+1967%2E+Depression%3A+Clinical%2C+experimental%2C+and+theoretical+aspects%2E+University+of+Pennsylvania+Press%2E)
6. Philip Bramsen, Martha Escobar-Molano, Ami Patel, and Rafael Alonso. 2011. Extracting social power relationships from natural language. In HLT-NAACL. [find](https://scholar.google.com/scholar?q=Philip+Bramsen%2C+Martha+Escobar%2DMolano%2C+Ami+Patel%2C+and+Rafael+Alonso%2E+2011%2E+Extracting+social+power+relationships+from+natural+language%2E+In+HLT%2DNAACL%2E)
7. Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. 2014. Concreteness ratings for 40 thousand generally known English word lemmas. Behavior Research Methods, 46(3). [find](https://scholar.google.com/scholar?q=Marc+Brysbaert%2C+Amy+Beth+Warriner%2C+and+Victor+Kuperman%2E+2014%2E+Concreteness+ratings+for+40+thousand+generally+known+English+word+lemmas%2E+Behavior+Research+Methods%2C+46%283%29%2E)
8. R. Sherlock Campbell and James W. Pennebaker. 2003. The secret life of pronouns: Flexibility in writing style and physical health. Psychological Science, 14(1). [find](https://scholar.google.com/scholar?q=R%2E+Sherlock+Campbell+and+James+W%2E+Pennebaker%2E+2003%2E+The+secret+life+of+pronouns%3A+Flexibility+in+writing+style+and+physical+health%2E+Psychological+Science%2C+14%281%29%2E)
9. Ge Chen. 2014. Visualizations for mental health topic models. Master’s thesis, MIT. [find](https://scholar.google.com/scholar?q=Ge+Chen%2E+2014%2E+Visualizations+for+mental+health+topic+models%2E+Master%E2%80%99s+thesis%2C+MIT%2E)
10. Cristian Danescu-Niculescu-Mizil, Lillian Lee, Bo Pang, and Jon Kleinberg. 2012. Echoes of power: Language effects and power differences in social interaction. In WWW. [find](https://scholar.google.com/scholar?q=Cristian+Danescu%2DNiculescu%2DMizil%2C+Lillian+Lee%2C+Bo+Pang%2C+and+Jon+Kleinberg%2E+2012%2E+Echoes+of+power%3A+Language+effects+and+power+differences+in+social+interaction%2E+In+WWW%2E)
11. Cristian Danescu-Niculescu-Mizil. 2012. A computational approach to linguistic style coordination. Ph.D. thesis, Cornell University. [find](https://scholar.google.com/scholar?q=Cristian+Danescu%2DNiculescu%2DMizil%2E+2012%2E+A+computational+approach+to+linguistic+style+coordination%2E+Ph%2ED%2E+thesis%2C+Cornell+University%2E)
12. Karthik Dinakar, Allison J.B. Chaney, Henry Lieberman, and David M. Blei. 2014a. Real-time topic models for crisis counseling. In KDD DSSG Workshop. [find](https://scholar.google.com/scholar?q=Karthik+Dinakar%2C+Allison+J%2EB%2E+Chaney%2C+Henry+Lieberman%2C+and+David+M%2E+Blei%2E+2014a%2E+Real%2Dtime+topic+models+for+crisis+counseling%2E+In+KDD+DSSG+Workshop%2E)
13. Karthik Dinakar, Emily Weinstein, Henry Lieberman, and Robert Selman. 2014b. Stacked generalization learning to analyze teenage distress. In ICWSM. [find](https://scholar.google.com/scholar?q=Karthik+Dinakar%2C+Emily+Weinstein%2C+Henry+Lieberman%2C+and+Robert+Selman%2E+2014b%2E+Stacked+generalization+learning+to+analyze+teenage+distress%2E+In+ICWSM%2E)
14. Karthik Dinakar, Jackie Chen, Henry Lieberman, Rosalind Picard, and Robert Filbin. 2015. Mixed-initiative real-time topic modeling & visualization for crisis counseling. In ACM ICIUI. [find](https://scholar.google.com/scholar?q=Karthik+Dinakar%2C+Jackie+Chen%2C+Henry+Lieberman%2C+Rosalind+Picard%2C+and+Robert+Filbin%2E+2015%2E+Mixed%2Dinitiative+real%2Dtime+topic+modeling+%26+visualization+for+crisis+counseling%2E+In+ACM+ICIUI%2E)
15. Marco Guerini, Alberto Pepe, and Bruno Lepri. 2012. Do linguistic style and readability of scientific abstracts affect their virality? In ICWSM. [find](https://scholar.google.com/scholar?q=Marco+Guerini%2C+Alberto+Pepe%2C+and+Bruno+Lepri%2E+2012%2E+Do+linguistic+style+and+readability+of+scientific+abstracts+affect+their+virality%3F+In+ICWSM%2E)
16. Shane Haberstroh, Thelma Duffey, Marcheta Evans, Robert Gee, and Heather Trepal. 2007. The experience of online counseling. Journal of Mental Health Counseling, 29(3). [find](https://scholar.google.com/scholar?q=Shane+Haberstroh%2C+Thelma+Duffey%2C+Marcheta+Evans%2C+Robert+Gee%2C+and+Heather+Trepal%2E+2007%2E+The+experience+of+online+counseling%2E+Journal+of+Mental+Health+Counseling%2C+29%283%29%2E)
17. Christine Howes, Matthew Purver, and Rose McCabe. 2014. Linguistic indicators of severity and progress in online text-based therapy for depression. CLPsych Workshop at ACL 2014. [find](https://scholar.google.com/scholar?q=Christine+Howes%2C+Matthew+Purver%2C+and+Rose+McCabe%2E+2014%2E+Linguistic+indicators+of+severity+and+progress+in+online+text%2Dbased+therapy+for+depression%2E+CLPsych+Workshop+at+ACL+2014%2E)
18. Rongyao Huang. 2015. Language use in teenage crisis intervention and the immediate outcome: A machine automated analysis of large scale text data. Master’s thesis, Columbia University. [find](https://scholar.google.com/scholar?q=Rongyao+Huang%2E+2015%2E+Language+use+in+teenage+crisis+intervention+and+the+immediate+outcome%3A+A+machine+automated+analysis+of+large+scale+text+data%2E+Master%E2%80%99s+thesis%2C+Columbia+University%2E)
19. C.J. Hutto and Eric Gilbert. 2014. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In ICWSM. [find](https://scholar.google.com/scholar?q=C%2EJ%2E+Hutto+and+Eric+Gilbert%2E+2014%2E+Vader%3A+A+parsimonious+rule%2Dbased+model+for+sentiment+analysis+of+social+media+text%2E+In+ICWSM%2E)
20. Thomas R. Insel. 2008. Assessing the economic costs of serious mental illness. The American Journal of Psychiatry, 165(6). [find](https://scholar.google.com/scholar?q=Thomas+R%2E+Insel%2E+2008%2E+Assessing+the+economic+costs+of+serious+mental+illness%2E+The+American+Journal+of+Psychiatry%2C+165%286%29%2E)
21. John Kalafat, Madelyn Gould, Jimmie Lou Harris Mun-fakh, and Marjorie Kleinman. 2007. An evaluation of crisis hotline outcomes part 1: Nonsuicidal crisis callers. Suicide and Life-threatening Behavior, 37(3). [find](https://scholar.google.com/scholar?q=John+Kalafat%2C+Madelyn+Gould%2C+Jimmie+Lou+Harris+Mun%2Dfakh%2C+and+Marjorie+Kleinman%2E+2007%2E+An+evaluation+of+crisis+hotline+outcomes+part+1%3A+Nonsuicidal+crisis+callers%2E+Suicide+and+Life%2Dthreatening+Behavior)
22. William Labov and David Fanshel. 1977. Therapeutic discourse: Psychotherapy as conversation. [find](https://scholar.google.com/scholar?q=William+Labov+and+David+Fanshel%2E+1977%2E+Therapeutic+discourse%3A+Psychotherapy+as+conversation%2E)
23. Dana Heller Levitt and Jodi D. Jacques. 2005. Promoting tolerance for ambiguity in counselor training programs. The Journal of Humanistic Counseling, Education and Development, 44(1). [find](https://scholar.google.com/scholar?q=Dana+Heller+Levitt+and+Jodi+D%2E+Jacques%2E+2005%2E+Promoting+tolerance+for+ambiguity+in+counselor+training+programs%2E+The+Journal+of+Humanistic+Counseling%2C+Education+and+Development%2C+44%281%29%2E)
24. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. 2013. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In NIPS.
25. National Institute of Mental Health. 2015. Any mental illness (AMI) among U.S. adults. [http://www.nimh.nih.gov/health/](http://www.nimh.nih.gov/health/) statistics/prevalence/any-mental-illness-ami-among-us-adults.shtml. Retrieved June 3, 2016. [↗](http://www.nimh.nih.gov/health/)
26. Kate G. Niederhoffer and James W. Pennebaker. 2002. Linguistic style matching in social interaction. Journal of Language and Social Psychology, 21(4). [find](https://scholar.google.com/scholar?q=Kate+G%2E+Niederhoffer+and+James+W%2E+Pennebaker%2E+2002%2E+Linguistic+style+matching+in+social+interaction%2E+Journal+of+Language+and+Social+Psychology%2C+21%284%29%2E)
27. Michael J. Paul. 2012. Mixed membership Markov models for unsupervised conversation modeling. In EMNLP-CoNLL. [find](https://scholar.google.com/scholar?q=Michael+J%2E+Paul%2E+2012%2E+Mixed+membership+Markov+models+for+unsupervised+conversation+modeling%2E+In+EMNLP%2DCoNLL%2E)
28. James W. Pennebaker, Matthias R. Mehl, and Kate G. Niederhoffer. 2003. Psychological aspects of natural language use: Our words, our selves. Annual Review of Psychology, 54(1). [find](https://scholar.google.com/scholar?q=James+W%2E+Pennebaker%2C+Matthias+R%2E+Mehl%2C+and+Kate+G%2E+Niederhoffer%2E+2003%2E+Psychological+aspects+of+natural+language+use%3A+Our+words%2C+our+selves%2E+Annual+Review+of+Psychology%2C+54%281%29%2E)
29. John P. Pestian, Pawel Matykiewicz, Michelle Linn-Gust, Brett South, Ozlem Uzuner, Jan Wiebe, Kevin B. Cohen, John Hurdle, and Christopher Brew. 2012. Sentiment analysis of suicide notes: A shared task. Biomedical informatics insights, 5(Suppl. 1). [find](https://scholar.google.com/scholar?q=John+P%2E+Pestian%2C+Pawel+Matykiewicz%2C+Michelle+Linn%2DGust%2C+Brett+South%2C+Ozlem+Uzuner%2C+Jan+Wiebe%2C+Kevin+B%2E+Cohen%2C+John+Hurdle%2C+and+Christopher+Brew%2E+2012%2E+Sentiment+analysis+of+suicide+notes%3A+A+shared+tas)
30. Tom Pyszczynski and Jeff Greenberg. 1987. Self-regulatory perseveration and the depressive self-focusing style: A self-awareness theory of reactive depression. Psychological Bulletin, 102(1). [find](https://scholar.google.com/scholar?q=Tom+Pyszczynski+and+Jeff+Greenberg%2E+1987%2E+Self%2Dregulatory+perseveration+and+the+depressive+self%2Dfocusing+style%3A+A+self%2Dawareness+theory+of+reactive+depression%2E+Psychological+Bulletin%2C+102%281%29%2E)
31. Tom Pyszczynski, Kathleen Holt, and Jeff Greenberg. 1987. Depression, self-focused attention, and expectancies for positive and negative future life events for self and others. Journal of Personality and Social Psychology, 52(5). [find](https://scholar.google.com/scholar?q=Tom+Pyszczynski%2C+Kathleen+Holt%2C+and+Jeff+Greenberg%2E+1987%2E+Depression%2C+self%2Dfocused+attention%2C+and+expectancies+for+positive+and+negative+future+life+events+for+self+and+others%2E+Journal+of+Personality+)
32. Nairan Ramirez-Esparza, Cindy Chung, Ewa Kacewicz, and James W. Pennebaker. 2008. The psychology of word use in depression forums in English and in Spanish: Testing two text analytic approaches. In ICWSM. [find](https://scholar.google.com/scholar?q=Nairan+Ramirez%2DEsparza%2C+Cindy+Chung%2C+Ewa+Kacewicz%2C+and+James+W%2E+Pennebaker%2E+2008%2E+The+psychology+of+word+use+in+depression+forums+in+English+and+in+Spanish%3A+Testing+two+text+analytic+approaches%2E+In+IC)
33. Shiquan Ren, Hong Lai, Wenjing Tong, Mostafa Amin-zadeh, Xuezhang Hou, and Shenghan Lai. 2010. Non-parametric bootstrapping for hierarchical data. Journal of Applied Statistics, 37(9). [find](https://scholar.google.com/scholar?q=Shiquan+Ren%2C+Hong+Lai%2C+Wenjing+Tong%2C+Mostafa+Amin%2Dzadeh%2C+Xuezhang+Hou%2C+and+Shenghan+Lai%2E+2010%2E+Non%2Dparametric+bootstrapping+for+hierarchical+data%2E+Journal+of+Applied+Statistics%2C+37%289%29%2E)
34. Alan Ritter, Colin Cherry, and Bill Dolan. 2010. [Unsupervised modeling of Twitter conversations](https://md.research.elamur.ai/free/en/papers/ritter-2010-unsupervised-modeling-twitter-conversations/). In HLT-NAACL.
35. Harvey Sacks and Gail Jefferson. 1995. Lectures on conversation. [find](https://scholar.google.com/scholar?q=Harvey+Sacks+and+Gail+Jefferson%2E+1995%2E+Lectures+on+conversation%2E)
36. Wiley-Blackwell. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. [Dialogue act modeling for automatic tagging and recognition of conversational speech](https://md.research.elamur.ai/free/en/papers/stolcke-2000-dialogue-act-modeling-automatic-tagging/). Computational Linguistics, 2…
37. Yla R. Tausczik and James W. Pennebaker. 2010. The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology, 29(1). [find](https://scholar.google.com/scholar?q=Yla+R%2E+Tausczik+and+James+W%2E+Pennebaker%2E+2010%2E+The+psychological+meaning+of+words%3A+LIWC+and+computerized+text+analysis+methods%2E+Journal+of+Language+and+Social+Psychology%2C+29%281%29%2E)
38. Teun Van Dijk. 1997. Discourse studies: A multidisciplinary approach. [find](https://scholar.google.com/scholar?q=Teun+Van+Dijk%2E+1997%2E+Discourse+studies%3A+A+multidisciplinary+approach%2E)
39. SAGE. World Health Organization. 2015. Depression: Fact sheet no 369. [http://www.who.int/](http://www.who.int/) mediacentre/factsheets/fs369/en/. Retrieved November 2, 2015. [↗](http://www.who.int/)
40. Jaewon Yang, Julian McAuley, Jure Leskovec, Paea LePendu, and Nigam Shah. 2014. Finding progression stages in time-evolving event sequences. In WWW. [find](https://scholar.google.com/scholar?q=Jaewon+Yang%2C+Julian+McAuley%2C+Jure+Leskovec%2C+Paea+LePendu%2C+and+Nigam+Shah%2E+2014%2E+Finding+progression+stages+in+time%2Devolving+event+sequences%2E+In+WWW%2E)

*Updated 24 October 2026*