---
title: "Expert, Crowdsourced, and Machine Assessment of — Shing 2018"
description: "The paper asks whether clinicians can reliably judge suicide risk from social media text alone, without talking to the person."
authors: ["Han-Chin Shing", "Suraj Nair", "Ayah Zirikly", "Meir Friedenberg", "Hal Daumé III", "Philip Resnik"]
year: 2018
venue: "Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic"
source: "https://aclanthology.org/W18-0603/"
doi: "10.18653/v1/w18-0603"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shing-2018-expert-crowdsourced-machine-assessment-suicide/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shing-2018-expert-crowdsourced-machine-assessment-suicide/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/) / Expert, Crowdsourced, and Machine Assessment of Suicide Risk via Online Postings

# Expert, Crowdsourced, and Machine Assessment of Suicide Risk via Online Postings

**Authors:** Han-Chin Shing, Suraj Nair, [Ayah Zirikly](https://md.research.elamur.ai/free/en/authors/ayah-zirikly/), Meir Friedenberg, [Hal Daumé III](https://md.research.elamur.ai/free/en/authors/hal-daume-iii/), [Philip Resnik](https://md.research.elamur.ai/free/en/authors/philip-resnik/) · **Year:** 2018 · **Venue:** Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W18-0603/) · [PDF](https://aclanthology.org/W18-0603.pdf) · [DOI 10.18653/v1/w18-0603](https://doi.org/10.18653/v1/w18-0603) · [OpenAlex](https://openalex.org/W2807452501)

**Citations:** 208 citations (OpenAlex, 2026-10-14) · [80](https://md.research.elamur.ai/free/en/papers/shing-2018-expert-crowdsourced-machine-assessment-suicide/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper asks whether clinicians can reliably judge suicide risk from social media text alone, without talking to the person. The authors built a dataset from a Reddit snapshot (2008–2015). It covers users who posted in the SuicideWatch subreddit, plus an equal number of control users who never posted in mental-health forums. Usernames were replaced with numeric IDs. Each user's SuicideWatch history was split into units of up to five posts. Crowd workers labeled these units on a four-level scale: None, Low, Moderate, Severe. Four suicide-assessment experts independently labeled a subset. Two of the experts used a brief category description, and two used a detailed rubric organized around families of risk factors. Expert agreement passed the usual reliability threshold, which the authors present as the first such evidence for social-media-based risk assessment. The detailed rubric produced more consistent judgments. The authors also compare crowd judgments with expert ones and report baseline machine classification experiments. The dataset will be distributed under governance by the American Association of Suicidology.

## Key points
- **Data scale:** 11,129 SuicideWatch users with ten or more Reddit posts, totaling about 1.56M posts. Of 1,097 sampled users, 934 were kept, yielding 982 annotation units. Full Reddit histories were extracted for these users and for the controls.
- **Risk scale:** four levels, adapted from Corbitt-Hall et al. (2016). A user's overall label is the highest risk assigned to any of their units.
- **Rubric:** the detailed version groups risk factors into four families:
  - thoughts, such as perceived burdensomeness;
  - feelings, such as hopelessness or agitation;
  - logistics, such as methods or access to lethal means;
  - context, such as prior attempts or isolation.
- **Expert agreement:** four experts labeled 250 units from 245 users. Mean pairwise Krippendorff's α was 0.812. The pair using short instructions agreed noticeably less (0.768).
- **Consensus and access:** consensus labels were inferred with the Dawid–Skene model. Researchers can request the data through AAS, with panel review covering IRB procedures and a ban on linking to other sites.

## Interesting ideas
- **Self-selection as a weak label.** Treating participation in a topical forum as a candidate-positive signal, with matched controls, gets around missing clinical ground truth. This could work for any condition that has its own online community.
- **Comparing instruction styles.** Giving annotators brief versus detailed instructions, then comparing agreement, directly tests whether a guideline adds value. Any annotation project could run this check before scaling up.
- **"Never downgrade" plus evidence selection.** Annotators keep the highest risk seen across posts and mark the single post that best supports their label. This makes the labels conservative and also produces post-level evidence that could later supervise attention or explanation models.
- **Capped annotation units.** Splitting long user histories into chunks of at most five posts keeps each item manageable for crowd workers. The highest chunk label then becomes the user-level label.
- **Gated release of sensitive data.** Releasing the dataset through a domain organization that reviews access requests offers a practical model for sharing sensitive text corpora responsibly.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shing2018expert,
  title = {{Expert, Crowdsourced, and Machine Assessment of Suicide Risk via Online Postings}},
  author = {Shing, Han-Chin and Nair, Suraj and Zirikly, Ayah and Friedenberg, Meir and III, Hal Daumé and Resnik, Philip},
  booktitle = {Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic},
  year = {2018},
  doi = {10.18653/v1/w18-0603},
  url = {https://doi.org/10.18653/v1/w18-0603}
}
```

- [← previous A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature](https://md.research.elamur.ai/free/en/papers/nye-2018-corpus-multi-level-annotations-patients/)
- [next → SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/wang-2018-switchout-efficient-data-augmentation-algorithm/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/)

## References

57 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jonathan Anderson. 1983. Lix and rix: Variations on a little-known readability index. Journal of Reading 26(6):490–496. [find](https://scholar.google.com/scholar?q=Jonathan+Anderson%2E+1983%2E+Lix+and+rix%3A+Variations+on+a+little%2Dknown+readability+index%2E+Journal+of+Reading+26%286%29%3A490%E2%80%93496%2E)
2. Philip J Batterham, Maria Ftanou, Jane Pirkis, Jacqueline L Brewer, Andrew J Mackinnon, Annette Beau-trais, A Kate Fairweather-Schmidt, and Helen Christensen. 2015. A systematic review and evaluation of measures for suicidal ideation and behaviors in Glen Coppersmith, Kim Ngo, Ryan Leary, and Anthon… [find](https://scholar.google.com/scholar?q=Philip+J+Batterham%2C+Maria+Ftanou%2C+Jane+Pirkis%2C+Jacqueline+L+Brewer%2C+Andrew+J+Mackinnon%2C+Annette+Beau%2Dtrais%2C+A+Kate+Fairweather%2DSchmidt%2C+and+Helen+Christensen%2E+2015%2E+A+systematic+review+and+evaluation+)
3. Adrian Benton, Glen Coppersmith, and Mark Dredze. 2017. Ethical research protocols for social media health research. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing. pages 94–102. [find](https://scholar.google.com/scholar?q=Adrian+Benton%2C+Glen+Coppersmith%2C+and+Mark+Dredze%2E+2017%2E+Ethical+research+protocols+for+social+media+health+research%2E+In+Proceedings+of+the+First+ACL+Workshop+on+Ethics+in+Natural+Language+Processing%2E+) [our card](https://md.research.elamur.ai/free/en/papers/benton-2017-ethical-research-protocols-social-media/)
4. Darcy J Corbitt-Hall, Jami M Gauthier, Margaret T Davis, and Tracy K Witte. 2016. College students’ responses to suicidal content on social networking sites: an examination using a simulated Facebook newsfeed. Suicide and life-threatening behavior 46(5):609–624. [find](https://scholar.google.com/scholar?q=Darcy+J+Corbitt%2DHall%2C+Jami+M+Gauthier%2C+Margaret+T+Davis%2C+and+Tracy+K+Witte%2E+2016%2E+College+students%E2%80%99+responses+to+suicidal+content+on+social+networking+sites%3A+an+examination+using+a+simulated+Facebook+)
5. Steven Bethard, Guergana Savova, Martha Palmer, and James Pustejovsky. 2017. Semeval-2017 task 12: Clinical tempeval. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). pages 565–572. [find](https://scholar.google.com/scholar?q=Steven+Bethard%2C+Guergana+Savova%2C+Martha+Palmer%2C+and+James+Pustejovsky%2E+2017%2E+Semeval%2D2017+task+12%3A+Clinical+tempeval%2E+In+Proceedings+of+the+11th+International+Workshop+on+Semantic+Evaluation+%28SemEval%2D)
6. Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied statistics pages 20–28. [find](https://scholar.google.com/scholar?q=Alexander+Philip+Dawid+and+Allan+M+Skene%2E+1979%2E+Maximum+likelihood+estimation+of+observer+error%2Drates+using+the+em+algorithm%2E+Applied+statistics+pages+20%E2%80%9328%2E)
7. David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent Dirichlet allocation. Journal of machine Learning research 3(Jan):993–1022. [find](https://scholar.google.com/scholar?q=David+M+Blei%2C+Andrew+Y+Ng%2C+and+Michael+I+Jordan%2E+2003%2E+Latent+Dirichlet+allocation%2E+Journal+of+machine+Learning+research+3%28Jan%29%3A993%E2%80%931022%2E) [our card](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/)
8. Munmun De Choudhury and Sushovan De. 2014. Mental health discourse on Reddit: Self-disclosure, social support, and anonymity. In ICWSM. [find](https://scholar.google.com/scholar?q=Munmun+De+Choudhury+and+Sushovan+De%2E+2014%2E+Mental+health+discourse+on+Reddit%3A+Self%2Ddisclosure%2C+social+support%2C+and+anonymity%2E+In+ICWSM%2E)
9. Bureau of Health Workforce. 2017. Designated health professional shortage areas: Statistics, first quarter of fiscal year 2018, designated HPSA quarterly summary. Health Resources and Services Administration (HRSA) U.S. Department of Health & Human Services, [https://ersrs.hrsa.gov/](https://ersrs.hrsa.gov/) ReportServer?/HGD… [↗](https://ersrs.hrsa.gov/)
10. Munmun De Choudhury, Emre Kiciman, Mark Dredze, Glen Coppersmith, and Mrinal Kumar. 2016. Discovering shifts to suicidal ideation from mental health content in social media. In Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, pages 2098–2110. [find](https://scholar.google.com/scholar?q=Munmun+De+Choudhury%2C+Emre+Kiciman%2C+Mark+Dredze%2C+Glen+Coppersmith%2C+and+Mrinal+Kumar%2E+2016%2E+Discovering+shifts+to+suicidal+ideation+from+mental+health+content+in+social+media%2E+In+Proceedings+of+the+2016)
11. Rafael A Calvo, David N Milne, M Sazzad Hussain, and Helen Christensen. 2017. Natural language processing in mental health applications using non-clinical texts. Natural Language Engineering 23(5):649–685. [find](https://scholar.google.com/scholar?q=Rafael+A+Calvo%2C+David+N+Milne%2C+M+Sazzad+Hussain%2C+and+Helen+Christensen%2E+2017%2E+Natural+language+processing+in+mental+health+applications+using+non%2Dclinical+texts%2E+Natural+Language+Engineering+23%285%29%3A649)
12. Katie G Egan, Rosalind N Koff, and Megan A Moreno. 2013. College students responses to mental health status updates on Facebook. Issues in mental health nursing 34(1):46–51. [find](https://scholar.google.com/scholar?q=Katie+G+Egan%2C+Rosalind+N+Koff%2C+and+Megan+A+Moreno%2E+2013%2E+College+students+responses+to+mental+health+status+updates+on+Facebook%2E+Issues+in+mental+health+nursing+34%281%29%3A46%E2%80%9351%2E)
13. James N Farr, James J Jenkins, and Donald G Paterson. 1951. Simplification of Flesch reading ease formula. Journal of applied psychology 35(5):333. [find](https://scholar.google.com/scholar?q=James+N+Farr%2C+James+J+Jenkins%2C+and+Donald+G+Paterson%2E+1951%2E+Simplification+of+Flesch+reading+ease+formula%2E+Journal+of+applied+psychology+35%285%29%3A333%2E)
14. Kimberly M Christopherson. 2007. The positive and negative implications of anonymity in internet social interactions:on the internet, nobody knows youre a dog. Computers in Human Behavior 23(6):3038– 3056. [find](https://scholar.google.com/scholar?q=Kimberly+M+Christopherson%2E+2007%2E+The+positive+and+negative+implications+of+anonymity+in+internet+social+interactions%3Aon+the+internet%2C+nobody+knows+youre+a+dog%2E+Computers+in+Human+Behavior+23%286%29%3A3038%E2%80%93+)
15. Ethan Fast, Binbin Chen, and Michael S Bernstein. 2016. Empath: Understanding topic signals in large-scale text. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, pages 4647–4657. [find](https://scholar.google.com/scholar?q=Ethan+Fast%2C+Binbin+Chen%2C+and+Michael+S+Bernstein%2E+2016%2E+Empath%3A+Understanding+topic+signals+in+large%2Dscale+text%2E+In+Proceedings+of+the+2016+CHI+Conference+on+Human+Factors+in+Computing+Systems%2E+ACM%2C+p)
16. Meri Coleman and Ta Lin Liau. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology 60(2):283. [find](https://scholar.google.com/scholar?q=Meri+Coleman+and+Ta+Lin+Liau%2E+1975%2E+A+computer+readability+formula+designed+for+machine+scoring%2E+Journal+of+Applied+Psychology+60%282%29%3A283%2E)
17. Lucie Flekova and Iryna Gurevych. 2016. Supersense embeddings: A unified model for supersense interpretation, prediction, and utilization. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). volume 1, pages 2029–2041. [find](https://scholar.google.com/scholar?q=Lucie+Flekova+and+Iryna+Gurevych%2E+2016%2E+Supersense+embeddings%3A+A+unified+model+for+supersense+interpretation%2C+prediction%2C+and+utilization%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+)
18. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research 12(Aug):2493–2537. [find](https://scholar.google.com/scholar?q=Ronan+Collobert%2C+Jason+Weston%2C+Le%CC%81on+Bottou%2C+Michael+Karlen%2C+Koray+Kavukcuoglu%2C+and+Pavel+Kuksa%2E+2011%2E+Natural+language+processing+%28almost%29+from+scratch%2E+Journal+of+Machine+Learning+Research+12%28Aug%29%3A2) [our card](https://md.research.elamur.ai/free/en/papers/collobert-2011-natural-language-processing-almost-scratch/)
19. Mike Conway and Daniel O’Connor. 2016. Social media, big data, and mental health: current advances and ethical implications. Current opinion in psychology 9:77–82. [find](https://scholar.google.com/scholar?q=Mike+Conway+and+Daniel+O%E2%80%99Connor%2E+2016%2E+Social+media%2C+big+data%2C+and+mental+health%3A+current+advances+and+ethical+implications%2E+Current+opinion+in+psychology+9%3A77%E2%80%9382%2E)
20. Rise B Goldstein, Donald W Black, Amelia Nasrallah, and George Winokur. 1991. The prediction of suicide: Sensitivity, specificity, and predictive value of a multivariate model applied to suicide among 1906 patients with affective disorders. Archives of general psychiatry 48(5):418–422. [find](https://scholar.google.com/scholar?q=Rise+B+Goldstein%2C+Donald+W+Black%2C+Amelia+Nasrallah%2C+and+George+Winokur%2E+1991%2E+The+prediction+of+suicide%3A+Sensitivity%2C+specificity%2C+and+predictive+value+of+a+multivariate+model+applied+to+suicide+among)
21. Glen Coppersmith, Mark Dredze, and Craig Harman. 2014. Quantifying mental health signals in Twitter. In Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality. pages 51–60. [find](https://scholar.google.com/scholar?q=Glen+Coppersmith%2C+Mark+Dredze%2C+and+Craig+Harman%2E+2014%2E+Quantifying+mental+health+signals+in+Twitter%2E+In+Proceedings+of+the+Workshop+on+Computational+Linguistics+and+Clinical+Psychology%3A+From+Linguisti) [our card](https://md.research.elamur.ai/free/en/papers/coppersmith-2014-quantifying-mental-health-signals-twitter/)
22. Kathleen M Griffiths, Louise Farrer, and Helen Christensen. 2010. The efficacy of internet interventions for depression and anxiety disorders: a review of randomised controlled trials. Medical Journal of Australia 192(11):S4. [find](https://scholar.google.com/scholar?q=Kathleen+M+Griffiths%2C+Louise+Farrer%2C+and+Helen+Christensen%2E+2010%2E+The+efficacy+of+internet+interventions+for+depression+and+anxiety+disorders%3A+a+review+of+randomised+controlled+trials%2E+Medical+Journal)
23. Glen Coppersmith, Ryan Leary, Eric Whyne, and Tony Wood. 2015. Quantifying suicidal ideation via language usage on social media. In Joint Statistics Meetings Proceedings, Statistical Computing Section, JSM. [find](https://scholar.google.com/scholar?q=Glen+Coppersmith%2C+Ryan+Leary%2C+Eric+Whyne%2C+and+Tony+Wood%2E+2015%2E+Quantifying+suicidal+ideation+via+language+usage+on+social+media%2E+In+Joint+Statistics+Meetings+Proceedings%2C+Statistical+Computing+Section)
24. Robert Gunning. 1952. The technique of clear writing . [find](https://scholar.google.com/scholar?q=Robert+Gunning%2E+1952%2E+The+technique+of+clear+writing+%2E)
25. Linguistics and Clinical Psychology. Association for Computational Linguistics, San Diego, CA, USA, pages 118–127. [http://www.aclweb.org/](http://www.aclweb.org/) anthology/W16-0312. [↗](http://www.aclweb.org/)
26. Sharath Chandra Guntuku, David B Yaden, Margaret L Kern, Lyle H Ungar, and Johannes C Eichstaedt. 2017. Detecting depression and mental illness on social media: an integrative review. Current Opinion in Behavioral Sciences 18:43–49. [find](https://scholar.google.com/scholar?q=Sharath+Chandra+Guntuku%2C+David+B+Yaden%2C+Margaret+L+Kern%2C+Lyle+H+Ungar%2C+and+Johannes+C+Eichstaedt%2E+2017%2E+Detecting+depression+and+mental+illness+on+social+media%3A+an+integrative+review%2E+Current+Opinion+)
27. D.N. Milne. 2017. Triaging content in online peer-support: an overview of the 2017 CLPsych shared task. Available online at [http://clpsych.org/shared-task-2017](http://clpsych.org/shared-task-2017). [↗](http://clpsych.org/shared-task-2017)
28. Christopher Homan, Ravdeep Johar, Tong Liu, Megan Lytle, Vincent Silenzio, and Cecilia Ovesdotter Alm. 2014. Toward macro-insights for suicide prevention: Analyzing fine-grained distress at scale. In Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Si… [find](https://scholar.google.com/scholar?q=Christopher+Homan%2C+Ravdeep+Johar%2C+Tong+Liu%2C+Megan+Lytle%2C+Vincent+Silenzio%2C+and+Cecilia+Ovesdotter+Alm%2E+2014%2E+Toward+macro%2Dinsights+for+suicide+prevention%3A+Analyzing+fine%2Dgrained+distress+at+scale%2E+In+)
29. Saif M Mohammad and Peter D Turney. 2013. Crowdsourcing a word–emotion association lexicon. Computational Intelligence 29(3):436–465. [find](https://scholar.google.com/scholar?q=Saif+M+Mohammad+and+Peter+D+Turney%2E+2013%2E+Crowdsourcing+a+word%E2%80%93emotion+association+lexicon%2E+Computational+Intelligence+29%283%29%3A436%E2%80%93465%2E)
30. Jr Thomas E Joiner, Rheeda L Walker, Jeremy W Pettit, Marisol Perez, and Kelly C Cukrowicz. 2005. Evidence-based assessment of depression in adults. Psychological Assessment 17(3):267. [find](https://scholar.google.com/scholar?q=Jr+Thomas+E+Joiner%2C+Rheeda+L+Walker%2C+Jeremy+W+Pettit%2C+Marisol+Perez%2C+and+Kelly+C+Cukrowicz%2E+2005%2E+Evidence%2Dbased+assessment+of+depression+in+adults%2E+Psychological+Assessment+17%283%29%3A267%2E)
31. Kevin A Padrez, Lyle Ungar, Hansen Andrew Schwartz, Robert J Smith, Shawndra Hill, Tadas Antanavicius, Dana M Brown, Patrick Crutchley, David A Asch, and Raina M Merchant. 2015. Linking social media and medical record data: a study of adults presenting to an academic, urban emergency department. BMJ… [find](https://scholar.google.com/scholar?q=Kevin+A+Padrez%2C+Lyle+Ungar%2C+Hansen+Andrew+Schwartz%2C+Robert+J+Smith%2C+Shawndra+Hill%2C+Tadas+Antanavicius%2C+Dana+M+Brown%2C+Patrick+Crutchley%2C+David+A+Asch%2C+and+Raina+M+Merchant%2E+2015%2E+Linking+social+media+a)
32. Jr Thomas E Joiner, Rheeda L Walker, M David Rudd, and David A Jobes. 1999. Scientizing and routinizing the assessment of suicidality in outpatient practice. Professional psychology: Research and practice 30(5):447. [find](https://scholar.google.com/scholar?q=Jr+Thomas+E+Joiner%2C+Rheeda+L+Walker%2C+M+David+Rudd%2C+and+David+A+Jobes%2E+1999%2E+Scientizing+and+routinizing+the+assessment+of+suicidality+in+outpatient+practice%2E+Professional+psychology%3A+Research+and+prac)
33. Rebecca J Passonneau and Bob Carpenter. 2014. The benefits of a model of annotation. Transactions of the Association for Computational Linguistics 2:311–326. [find](https://scholar.google.com/scholar?q=Rebecca+J+Passonneau+and+Bob+Carpenter%2E+2014%2E+The+benefits+of+a+model+of+annotation%2E+Transactions+of+the+Association+for+Computational+Linguistics+2%3A311%E2%80%93326%2E) [our card](https://md.research.elamur.ai/free/en/papers/passonneau-2014-benefits-model-annotation/)
34. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 . [find](https://scholar.google.com/scholar?q=Yoon+Kim%2E+2014%2E+Convolutional+neural+networks+for+sentence+classification%2E+arXiv+preprint+arXiv%3A1408%2E5882+%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/)
35. Umashanthi Pavalanathan and Munmun De Choudhury. 2015. Identity management and mental health discourse in social media. In Proceedings of the 24th International Conference on World Wide Web. ACM, pages 315–321. [find](https://scholar.google.com/scholar?q=Umashanthi+Pavalanathan+and+Munmun+De+Choudhury%2E+2015%2E+Identity+management+and+mental+health+discourse+in+social+media%2E+In+Proceedings+of+the+24th+International+Conference+on+World+Wide+Web%2E+ACM%2C+page)
36. J Peter Kincaid, Jr Robert P Fishburne, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, Naval Technical Training Command Millington TN Research B… [find](https://scholar.google.com/scholar?q=J+Peter+Kincaid%2C+Jr+Robert+P+Fishburne%2C+Richard+L+Rogers%2C+and+Brad+S+Chissom%2E+1975%2E+Derivation+of+new+readability+formulas+%28automated+readability+index%2C+fog+count+and+flesch+reading+ease+formula%29+for+)
37. Bryan Klimt and Yiming Yang. 2004. The enron corpus: A new dataset for email classification research. In European Conference on Machine Learning. Springer, pages 217–226. [find](https://scholar.google.com/scholar?q=Bryan+Klimt+and+Yiming+Yang%2E+2004%2E+The+enron+corpus%3A+A+new+dataset+for+email+classification+research%2E+In+European+Conference+on+Machine+Learning%2E+Springer%2C+pages+217%E2%80%93226%2E)
38. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12:2825… [find](https://scholar.google.com/scholar?q=F%2E+Pedregosa%2C+G%2E+Varoquaux%2C+A%2E+Gramfort%2C+V%2E+Michel%2C+B%2E+Thirion%2C+O%2E+Grisel%2C+M%2E+Blondel%2C+P%2E+Prettenhofer%2C+R%2E+Weiss%2C+V%2E+Dubourg%2C+J%2E+Vanderplas%2C+A%2E+Passos%2C+D%2E+Cournapeau%2C+M%2E+Brucher%2C+M%2E+Perrot%2C+and+E%2E+Duc)
39. Klaus Krippendorff. 2004. Reliability in content analysis. Human communication research 30(3):411– 433. [find](https://scholar.google.com/scholar?q=Klaus+Krippendorff%2E+2004%2E+Reliability+in+content+analysis%2E+Human+communication+research+30%283%29%3A411%E2%80%93+433%2E)
40. Alex D Pokorny. 1983. Prediction of suicide in psychiatric patients: report of a prospective study. Archives of general psychiatry 40(3):249–257. [find](https://scholar.google.com/scholar?q=Alex+D+Pokorny%2E+1983%2E+Prediction+of+suicide+in+psychiatric+patients%3A+report+of+a+prospective+study%2E+Archives+of+general+psychiatry+40%283%29%3A249%E2%80%93257%2E)
41. Fabienne Lind, Maria Gruber, and Hajo G Boomgaarden. 2017. Content analysis by the crowd: Assessing the usability of crowdsourcing for coding latent constructs. Communication methods and measures 11(3):191–209. [find](https://scholar.google.com/scholar?q=Fabienne+Lind%2C+Maria+Gruber%2C+and+Hajo+G+Boomgaarden%2E+2017%2E+Content+analysis+by+the+crowd%3A+Assessing+the+usability+of+crowdsourcing+for+coding+latent+constructs%2E+Communication+methods+and+measures+11%283)
42. Dina Popovic, Eduard Vieta, Jean-Michel Azorin, Jules Angst, Charles L Bowden, Sergey Mosolov, Allan H Young, and Giulio Perugi. 2015. Suicide attempts in major depressive episode: evidence from the bridge-ii-mix study. Bipolar disorders 17(7):795–803. [find](https://scholar.google.com/scholar?q=Dina+Popovic%2C+Eduard+Vieta%2C+Jean%2DMichel+Azorin%2C+Jules+Angst%2C+Charles+L+Bowden%2C+Sergey+Mosolov%2C+Allan+H+Young%2C+and+Giulio+Perugi%2E+2015%2E+Suicide+attempts+in+major+depressive+episode%3A+evidence+from+the+b)
43. Tong Liu, Qijin Cheng, Christopher M Homan, and Vincent Silenzio. 2017. Learning from various labeling strategies for suicide-related messages on social media: An experimental study. ACM International Conference on Web Search and Data Mining Workshop on Mining Online Health Reports . [find](https://scholar.google.com/scholar?q=Tong+Liu%2C+Qijin+Cheng%2C+Christopher+M+Homan%2C+and+Vincent+Silenzio%2E+2017%2E+Learning+from+various+labeling+strategies+for+suicide%2Drelated+messages+on+social+media%3A+An+experimental+study%2E+ACM+International)
44. Reddit. 2018. Reddit privacy policy. Downloaded March 22, 2018, [https://www.reddit.com/](https://www.reddit.com/) help/privacypolicy/. [↗](https://www.reddit.com/)
45. G Harry Mc Laughlin. 1969. Smog grading-a new readability formula. Journal of reading 12(8):639– 646. [find](https://scholar.google.com/scholar?q=G+Harry+Mc+Laughlin%2E+1969%2E+Smog+grading%2Da+new+readability+formula%2E+Journal+of+reading+12%288%29%3A639%E2%80%93+646%2E)
46. Philip Resnik. 2017. The (in)ability to triangulate in data driven healthcare research. Presentation, SBS Decadal Survey - Workshop on Culture, Language, and Behavior, National Academies of Sciences, Engineering, and Medicine. [find](https://scholar.google.com/scholar?q=Philip+Resnik%2E+2017%2E+The+%28in%29ability+to+triangulate+in+data+driven+healthcare+research%2E+Presentation%2C+SBS+Decadal+Survey+%2D+Workshop+on+Culture%2C+Language%2C+and+Behavior%2C+National+Academies+of+Sciences%2C+)
47. David N. Milne, Glen Pink, Ben Hachey, and Rafael A. Calvo. 2016. [find](https://scholar.google.com/scholar?q=David+N%2E+Milne%2C+Glen+Pink%2C+Ben+Hachey%2C+and+Rafael+A%2E+Calvo%2E+2016%2E)
48. CLPsych 2016 shared task: Triaging content in online peer-support forums. In Proceedings of the Third Workshop on Computational Philip Resnik, William Armstrong, Leonardo Claudino, and Thang Nguyen. 2015. The university of maryland clpsych 2015 shared task system. In CLPsych@ HLT-NAACL. pages 54–60. [find](https://scholar.google.com/scholar?q=CLPsych+2016+shared+task%3A+Triaging+content+in+online+peer%2Dsupport+forums%2E+In+Proceedings+of+the+Third+Workshop+on+Computational+Philip+Resnik%2C+William+Armstrong%2C+Leonardo+Claudino%2C+and+Thang+Nguyen%2E+2)
49. Philip Resnik, Rebecca Resnik, and Margaret Mitchell, editors. 2014. Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality. Association for Computational Linguistics, Baltimore, Maryland, USA. [http://www.aclweb.org/](http://www.aclweb.org/) anthology/W/W… [↗](http://www.aclweb.org/)
50. Rebecca Resnik. 2016. Psychological assessment: The not good enough state of the art. Presentation, Veterans Affairs Suicide Prevention Innovations Conference (VASPI). [find](https://scholar.google.com/scholar?q=Rebecca+Resnik%2E+2016%2E+Psychological+assessment%3A+The+not+good+enough+state+of+the+art%2E+Presentation%2C+Veterans+Affairs+Suicide+Prevention+Innovations+Conference+%28VASPI%29%2E)
51. RJ Senter and Edgar A Smith. 1967. Automated readability index. Technical report, CINCINNATI UNIV OH. [find](https://scholar.google.com/scholar?q=RJ+Senter+and+Edgar+A+Smith%2E+1967%2E+Automated+readability+index%2E+Technical+report%2C+CINCINNATI+UNIV+OH%2E)
52. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y Ng. 2008. Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the conference on empirical methods in natural language processing. Association for Computational Linguistics, pages 25… [find](https://scholar.google.com/scholar?q=Rion+Snow%2C+Brendan+O%E2%80%99Connor%2C+Daniel+Jurafsky%2C+and+Andrew+Y+Ng%2E+2008%2E+Cheap+and+fast%E2%80%94but+is+it+good%3F%3A+evaluating+non%2Dexpert+annotations+for+natural+language+tasks%2E+In+Proceedings+of+the+conference+on+e) [our card](https://md.research.elamur.ai/free/en/papers/snow-2008-cheap-fast-but-is-it/)
53. Yla R Tausczik and James W Pennebaker. 2010. The psychological meaning of words: LIWC and computerized text analysis methods. Journal of language and social psychology 29(1):24–54. [find](https://scholar.google.com/scholar?q=Yla+R+Tausczik+and+James+W+Pennebaker%2E+2010%2E+The+psychological+meaning+of+words%3A+LIWC+and+computerized+text+analysis+methods%2E+Journal+of+language+and+social+psychology+29%281%29%3A24%E2%80%9354%2E)
54. M Johnson Vioulès, Bilel Moulahi, Jérôme Azé, and Sandra Bringay. 2018. Detection of suicide-related posts in twitter data streams. IBM Journal of Research and Development 62(1):7–1. [find](https://scholar.google.com/scholar?q=M+Johnson+Vioule%CC%80s%2C+Bilel+Moulahi%2C+Je%CC%81ro%CC%82me+Aze%CC%81%2C+and+Sandra+Bringay%2E+2018%2E+Detection+of+suicide%2Drelated+posts+in+twitter+data+streams%2E+IBM+Journal+of+Research+and+Development+62%281%29%3A7%E2%80%931%2E)
55. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 14… [find](https://scholar.google.com/scholar?q=Zichao+Yang%2C+Diyi+Yang%2C+Chris+Dyer%2C+Xiaodong+He%2C+Alex+Smola%2C+and+Eduard+Hovy%2E+2016%2E+Hierarchical+attention+networks+for+document+classification%2E+In+Proceedings+of+the+2016+Conference+of+the+North+Amer) [our card](https://md.research.elamur.ai/free/en/papers/yang-2016-hierarchical-attention-networks-document-classification/)
56. Andrew Yates, Arman Cohan, and Nazli Goharian. 2017. Depression and self-harm risk assessment in online forums. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, pages 2968–2978. [https://www.aclw…](https://www.aclw…) [↗](https://www.aclw…) [our card](https://md.research.elamur.ai/free/en/papers/yates-2017-depression-self-harm-risk-assessment/)
57. Ayah Zirikly, Varun Kumar, and Philip Resnik. 2016. The GW/UMD CLPsych 2016 shared task system. In CLPsych@ HLT-NAACL. pages 166–170. [find](https://scholar.google.com/scholar?q=Ayah+Zirikly%2C+Varun+Kumar%2C+and+Philip+Resnik%2E+2016%2E+The+GW%2FUMD+CLPsych+2016+shared+task+system%2E+In+CLPsych%40+HLT%2DNAACL%2E+pages+166%E2%80%93170%2E)