--- title: "Evaluating Document-Tuned Transformer Representations for" description: "The authors compare two models that share the same architecture: roberta-large (base) and all-roberta-large-v1 (document-tuned). 1 reference." authors: ["Aaron Marker", "Oscar Kjell", "Vasudha Varadarajan", "H. Andrew Schwartz"] year: 2026 venue: "Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2026)" source: "https://aclanthology.org/2026.clpsych-1.14/" doi: "10.18653/v1/2026.clpsych-1.14" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/marker-2026-evaluating-document-tuned-transformer-representations/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/marker-2026-evaluating-document-tuned-transformer-representations/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/12/) / Evaluating Document-Tuned Transformer Representations for Person-level Mental Health Assessment # Evaluating Document-Tuned Transformer Representations for Person-level Mental Health Assessment **Authors:** Aaron Marker, Oscar Kjell, [Vasudha Varadarajan](https://md.research.elamur.ai/free/en/authors/vasudha-varadarajan/) and [H. Andrew Schwartz](https://md.research.elamur.ai/free/en/authors/hansen-andrew-schwartz/) · **Year:** 2026 · **Venue:** [Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2026)](https://md.research.elamur.ai/free/en/venues/clpsych/) ([Workshop on Computational Linguistics and Clinical Psychology 2026](https://md.research.elamur.ai/free/en/venues/clpsych/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.clpsych-1.14/) · [PDF](https://aclanthology.org/2026.clpsych-1.14.pdf) · [DOI 10.18653/v1/2026.clpsych-1.14](https://doi.org/10.18653/v1/2026.clpsych-1.14) · [OpenAlex](https://openalex.org/W7166810632) **Citations:** 1 citations (OpenAlex, 2026-10-14) ## Summary The paper asks whether contrastively fine-tuned "sentence transformer" encoders produce better person-level representations for mental health assessment than standard masked-language-model encoders. Psychological outcomes are measured per person, but they must be inferred from many short texts. The authors compare two models that share the same architecture: roberta-large (base) and all-roberta-large-v1 (document-tuned). Embeddings are frozen, extracted layer by layer, aggregated across each person's messages by mean pooling or concatenation, and fed to the same downstream predictor. Accuracy is scored with Pearson r. Data come from two longitudinal EMA/survey collections, DS4UD and LMHD, covering depression, anxiety, stress, alcohol use, personality and related scales. Document-tuned embeddings were more accurate overall, by about 13.4% in Pearson r. The gain held across layers, datasets and outcome domains. Probes of word-level associations and noise perturbations did not explain the advantage. The authors therefore suggest document-tuned encoders as a reasonable default for language-based screening, while cautioning that better prediction does not imply clinical validity. ## Key points - **Controlled model pair:** roberta-large versus all-roberta-large-v1. The document-tuned model was further trained on roughly one billion sentence pairs, so the comparison is not fully matched on training data. - **Datasets:** - DS4UD: 120 U.S. service-industry workers, 10,108 EMAs, about 84.9 per person. - LMHD: 1,307 Prolific participants, 7,207 responses, about 5.5 per person; the sample was enriched with people reporting prior MDD or GAD diagnoses. - **Overall result:** document-tuned models improved Pearson r by 13.4% (p = .015). - On LMHD, mean pooling gave 0.572 versus 0.561 for the base model. - On DS4UD, concatenation gave 0.482 versus 0.411. - All of these differences were significant. - **Layer choice:** document-tuned models performed best in later layers, with the last four to five layers roughly equivalent. - **Linguistic analysis:** hedged words (e.g., "usually", "pretty") carried more outcome signal in document-tuned embeddings, while abundance words (e.g., "lot") carried more in base embeddings. - **Perturbation tests (deletion, typos, WordNet synonyms, back translation):** the abstract claims document-tuned models were more robust, but the results section reports mostly comparable degradation for both models. The base model was slightly more robust under heavy synonym replacement. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 539 works · 179,533 citations - [Lund University](https://md.research.elamur.ai/free/en/universities/lund-university/) — University, Lund, Sweden · 11 works · 1,263 citations - [Vanderbilt University](https://md.research.elamur.ai/free/en/universities/vanderbilt-university/) — University, Nashville, United States · 14 works · 1,413 citations 📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Social Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/social-psychology/) · 306 works 🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works ## Cite Cite as: Aaron Marker, Oscar Kjell, Vasudha Varadarajan and H. Andrew Schwartz. 2026. Evaluating Document-Tuned Transformer Representations for Person-level Mental Health Assessment. Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2026). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{marker2026evaluating, title = {{Evaluating Document-Tuned Transformer Representations for Person-level Mental Health Assessment}}, author = {Aaron Marker, Oscar Kjell, Vasudha Varadarajan and H. Andrew Schwartz}, booktitle = {Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2026)}, year = {2026}, doi = {10.18653/v1/2026.clpsych-1.14}, url = {https://doi.org/10.18653/v1/2026.clpsych-1.14} } ``` - [← previous Evaluating Answer Leakage Robustness of LLM Tutors against Adversarial Student Attacks](https://md.research.elamur.ai/free/en/papers/zhao-2026-evaluating-answer-leakage-robustness-llm/) - [next → Evaluating Encoder- and LLM-Based Approaches for Robust Indirect Personal Identifier Detection](https://md.research.elamur.ai/free/en/papers/otto-2026-evaluating-encoder-llm-based-approaches/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/12/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies,… *Updated 31 October 2026*