---
title: "Reading Between the Lines — paper by Soni et al. (2026)"
description: "Messages that patients send through portals tend to bury the actual clinical question in long, emotional… 61.0% ROUGE-L. 26 references."
authors: ["Sarvesh Soni", "Madeline Bittner", "Dina Demner-Fushman"]
year: 2026
venue: "BioNLP 2026"
source: "https://aclanthology.org/2026.bionlp-1.38/"
doi: "10.18653/v1/2026.bionlp-1.38"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/soni-2026-reading-between-lines-translating-verbose/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/soni-2026-reading-between-lines-translating-verbose/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/40/) / Reading Between the Lines: Toward Translating Verbose Patient-authored Messages into Clinician-Formulated Questions

# Reading Between the Lines: Toward Translating Verbose Patient-authored Messages into Clinician-Formulated Questions

**Authors:** [Sarvesh Soni](https://md.research.elamur.ai/free/en/authors/sarvesh-soni/), Madeline Bittner and [Dina Demner-Fushman](https://md.research.elamur.ai/free/en/authors/dina-demner-fushman/) · **Year:** 2026 · **Venue:** [BioNLP 2026](https://md.research.elamur.ai/free/en/venues/bionlp/) ([BioNLP 2026](https://md.research.elamur.ai/free/en/venues/bionlp/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.bionlp-1.38/) · [PDF](https://aclanthology.org/2026.bionlp-1.38.pdf) · [DOI 10.18653/v1/2026.bionlp-1.38](https://doi.org/10.18653/v1/2026.bionlp-1.38) · [OpenAlex](https://openalex.org/W7166828398)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Messages that patients send through portals tend to bury the actual clinical question in long, emotional stories, so clinicians have to work out what is really being asked. The authors frame this as a rewriting task. A patient narrative becomes a short question in a clinician's voice, phrased as a query to an EHR. Their pipeline is deliberately minimal. Llama 3.1 8B or Llama 3.3 70B is prompted to write one question of 10–15 words. A rule-based format check follows, and the model regenerates if the check fails, up to five tries. The pipeline uses no retrieval or external knowledge.

They tested it on 140 patient questions from ArchEHR-QA and its 2026 shared task. Two expert annotators labelled every output for quality and error type. The outputs mostly respected the format constraints, but they were frequently too generic or off-target, and some invented clinical details. Of ROUGE-L, BERTScore, Medcon and AlignScore, no metric cleanly separated the human quality levels. BERTScore agreed best with human pairwise preferences. The study highlights that expert evaluation remains necessary and that benchmarks need multiple references. The code and annotations are publicly released.

## Key points
- **Data:** 140 distinct questions were evaluated. 108 come from the original ArchEHR-QA release, which links forum questions to MIMIC notes, and 32 from the ArchEHR-QA 2026 Shared Task. Patient narratives average 100.4 words, and reference clinician questions average 11.9 words.
- **Length control:** The 70B model kept 95% of its outputs inside the 10–15-word range, averaging 11.5 words. The 8B model did so for only 48.6% of outputs, averaging 15.5 words.
- **Human evaluation:**
  - All 280 outputs were double-annotated with a quality label (Good/Ok/Bad) and error labels (Generic, Malformed, Tangential, Hallucination).
  - Agreement on quality was low: 36.8% for three classes (κ 0.162) and 71.1% when collapsed to binary (κ 0.234).
- **Error profile:**
  - Generic errors dominated with 236 instances, followed by Tangential (146), Hallucination (61) and Malformed (57).
  - The 70B model produced fewer Bad outputs. The 8B model hallucinated more often.
- **Metric meta-evaluation:** Pairwise accuracy was measured against the reference clinician question. BERTScore led at 61.0%, ROUGE-L and AlignScore stayed below 50%, and Medcon reached only 26.8%.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National Institutes of Health](https://md.research.elamur.ai/free/en/government/national-institutes-of-health/) — Government body, Bethesda, United States · 12 works · 3,846 citations
- [United States National Library of Medicine](https://md.research.elamur.ai/free/en/labs/united-states-national-library-of-medicine/) — Research institute, Bethesda, United States · 7 works · 5,342 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🩺 Medicine](https://md.research.elamur.ai/free/en/shelves/medicine/) · 241 works · [🩺 Health Informatics](https://md.research.elamur.ai/free/en/shelves/medicine/health-informatics/) · 184 works · [🏥 Health Professions](https://md.research.elamur.ai/free/en/shelves/health-professions/) · 69 works · [🏥 Health Information Management](https://md.research.elamur.ai/free/en/shelves/health-professions/health-information-management/) · 50 works

## Cite

Cite as: Sarvesh Soni, Madeline Bittner and Dina Demner-Fushman. 2026. Reading Between the Lines: Toward Translating Verbose Patient-authored Messages into Clinician-Formulated Questions. BioNLP 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{soni2026reading,
  title = {{Reading Between the Lines: Toward Translating Verbose Patient-authored Messages into Clinician-Formulated Questions}},
  author = {Sarvesh Soni, Madeline Bittner and Dina Demner-Fushman},
  booktitle = {BioNLP 2026},
  year = {2026},
  doi = {10.18653/v1/2026.bionlp-1.38},
  url = {https://doi.org/10.18653/v1/2026.bionlp-1.38}
}
```

- [← previous Read As Human: Compressing Context via Parallelizable Close Reading and Skimming](https://md.research.elamur.ai/free/en/papers/tang-2026-read-as-human-compressing-context/)
- [next → Reading the News: Adapting Large Language Models to Swedish Journalism Through Continued Pre-Training](https://md.research.elamur.ai/free/en/papers/borggren-2026-reading-news-adapting-large-language/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/40/)

## References

26 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Blake J. Anderson, Muhammad Zia ul Haq, Yuanda Zhu, Andrew Hornback, Alison D. Cowan, Michelle Mott, Bradley Gallaher, and Arash Harzand. 2025. Development and Evaluation of a Model to Manage Patient Portal Messages. NEJM AI, 2(3):AIoa2400354. [find](https://scholar.google.com/scholar?q=Blake+J%2E+Anderson%2C+Muhammad+Zia+ul+Haq%2C+Yuanda+Zhu%2C+Andrew+Hornback%2C+Alison+D%2E+Cowan%2C+Michelle+Mott%2C+Bradley+Gallaher%2C+and+Arash+Harzand%2E+2025%2E+Development+and+Evaluation+of+a+Model+to+Manage+Patient+)
2. Jayetri Bardhan, Kirk Roberts, and Daisy Zhe Wang. 2024. Question Answering for Electronic Health Records: Scoping Review of Datasets and Models. Journal of Medical Internet Research, 26(1):e53636. [find](https://scholar.google.com/scholar?q=Jayetri+Bardhan%2C+Kirk+Roberts%2C+and+Daisy+Zhe+Wang%2E+2024%2E+Question+Answering+for+Electronic+Health+Records%3A+Scoping+Review+of+Datasets+and+Models%2E+Journal+of+Medical+Internet+Research%2C+26%281%29%3Ae53636%2E)
3. Asma Ben Abacha, Yassine Mrabet, Yuhao Zhang, Chaitanya Shivade, Curtis Langlotz, and Dina Demner-Fushman. 2021. Overview of the MEDIQA 2021 Shared Task on Summarization in the Medical Domain. In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 74–85, Online. Association for… [find](https://scholar.google.com/scholar?q=Asma+Ben+Abacha%2C+Yassine+Mrabet%2C+Yuhao+Zhang%2C+Chaitanya+Shivade%2C+Curtis+Langlotz%2C+and+Dina+Demner%2DFushman%2E+2021%2E+Overview+of+the+MEDIQA+2021+Shared+Task+on+Summarization+in+the+Medical+Domain%2E+In+Proc)
4. Joshua M. Biro, Jessica L. Handley, J. Malcolm Mc-Curry, Adam Visconti, Jeffrey Weinfeld, J. Gregory Trafton, and Raj M. Ratwani. 2025. Opportunities and risks of artificial intelligence in patient portal messaging in primary care. npj Digital Medicine, 8(1):1–6. [find](https://scholar.google.com/scholar?q=Joshua+M%2E+Biro%2C+Jessica+L%2E+Handley%2C+J%2E+Malcolm+Mc%2DCurry%2C+Adam+Visconti%2C+Jeffrey+Weinfeld%2C+J%2E+Gregory+Trafton%2C+and+Raj+M%2E+Ratwani%2E+2025%2E+Opportunities+and+risks+of+artificial+intelligence+in+patient+po)
5. Shan Chen, Marco Guevara, Shalini Moningi, Frank Hoebers, Hesham Elhalawani, Benjamin H. Kann, Fallon E. Chipidza, Jonathan Leeman, Hugo J. W. L. Aerts, Timothy Miller, Guergana K. Savova, Jack Gallifant, Leo A. Celi, Raymond H. Mak, Maryam Lustberg, Majid Afshar, and Danielle S. Bitterman. 2024. Th… [find](https://scholar.google.com/scholar?q=Shan+Chen%2C+Marco+Guevara%2C+Shalini+Moningi%2C+Frank+Hoebers%2C+Hesham+Elhalawani%2C+Benjamin+H%2E+Kann%2C+Fallon+E%2E+Chipidza%2C+Jonathan+Leeman%2C+Hugo+J%2E+W%2E+L%2E+Aerts%2C+Timothy+Miller%2C+Guergana+K%2E+Savova%2C+Jack+Gallif)
6. Christine Dymek, Bryan Kim, Genevieve B Melton, Thomas H Payne, Hardeep Singh, and Chun-Ju Hsiao. 2021. Building the evidence-base to reduce electronic health record–related clinician burden. Journal of the American Medical Informatics Association, 28(5):1057–1061. [find](https://scholar.google.com/scholar?q=Christine+Dymek%2C+Bryan+Kim%2C+Genevieve+B+Melton%2C+Thomas+H+Payne%2C+Hardeep+Singh%2C+and+Chun%2DJu+Hsiao%2E+2021%2E+Building+the+evidence%2Dbase+to+reduce+electronic+health+record%E2%80%93related+clinician+burden%2E+Journal+)
7. Fernando Fernandez-Llimos. 2025. Consistency of Medical Subject Headings assignment: A test-retest reliability analysis. Research in Social and Administrative Pharmacy, 21(10):784–789. [find](https://scholar.google.com/scholar?q=Fernando+Fernandez%2DLlimos%2E+2025%2E+Consistency+of+Medical+Subject+Headings+assignment%3A+A+test%2Dretest+reliability+analysis%2E+Research+in+Social+and+Administrative+Pharmacy%2C+21%2810%29%3A784%E2%80%93789%2E)
8. Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frederic Blain, Tom Kocmi, Jiayi Wang, David Ifeoluwa Adelani, Marianna Buchicchio, Chrysoula Zerva, and Alon Lavie. 2024. [Are LLMs Breaking MT Metrics?](https://md.research.elamur.ai/free/en/papers/freitag-2024-are-llms-breaking-mt-metrics/) Results of the WMT24 Metrics Shared…
9. Patricia Garcia, Stephen P. Ma, Shreya Shah, Margaret Smith, Yejin Jeong, Anna Devon-Sand, Ming Tai-Seale, Kevin Takazawa, Danyelle Clutter, Kyle Vogt, Carlene Lugtu, Matthew Rojo, Steven Lin, Tait Shanafelt, Michael A. Pfeffer, and Christopher Sharp. 2024. Artificial Intelligence–Generated Draft Re… [find](https://scholar.google.com/scholar?q=Patricia+Garcia%2C+Stephen+P%2E+Ma%2C+Shreya+Shah%2C+Margaret+Smith%2C+Yejin+Jeong%2C+Anna+Devon%2DSand%2C+Ming+Tai%2DSeale%2C+Kevin+Takazawa%2C+Danyelle+Clutter%2C+Kyle+Vogt%2C+Carlene+Lugtu%2C+Matthew+Rojo%2C+Steven+Lin%2C+Tait+Sh)
10. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
11. Alistair E. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Liwei H. Lehman, Leo A. Celi, and Roger G. Mark. 2023. [MIMIC-IV, a freely accessible electronic health record dataset](https://md.research.elamur.ai/free/en/papers/johnson-2023-mimic-iv-freely-accessible-electronic/). Scientific Data, 10(1):1.
12. Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, Liwei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. 2016. [MIMIC-III, a freely accessible critical care database](https://md.research.elamur.ai/free/en/papers/johnson-2016-mimic-iii-freely-accessible-critical/). Scientific Data, 3(1):160035.
13. Chin-Yew Lin. 2004. [ROUGE: A Package for Automatic Evaluation of Summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
14. Francesco Moramarco, Alex Papadopoulos Korfiatis, Mark Perera, Damir Juric, Jack Flann, Ehud Reiter, Anya Belz, and Aleksandar Savkov. 2022. Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation. In Proceedings of the 60th Annual Meeting of the Association for Compu… [find](https://scholar.google.com/scholar?q=Francesco+Moramarco%2C+Alex+Papadopoulos+Korfiatis%2C+Mark+Perera%2C+Damir+Juric%2C+Jack+Flann%2C+Ehud+Reiter%2C+Anya+Belz%2C+and+Aleksandar+Savkov%2E+2022%2E+Human+Evaluation+and+Correlation+with+Automatic+Metrics+in+)
15. Victor Novack, Lone S. Avnon, Alexander Smolyakov, Rachel Barnea, Alan Jotkowitz, and Francisc Schlaef-fer. 2006. Disagreement in the interpretation of chest radiographs among specialists and clinical outcomes of patients hospitalized with suspected pneumonia. European Journal of Internal Medicine,… [find](https://scholar.google.com/scholar?q=Victor+Novack%2C+Lone+S%2E+Avnon%2C+Alexander+Smolyakov%2C+Rachel+Barnea%2C+Alan+Jotkowitz%2C+and+Francisc+Schlaef%2Dfer%2E+2006%2E+Disagreement+in+the+interpretation+of+chest+radiographs+among+specialists+and+clinical)
16. Yang Ren, Dezhi Wu, Aditya Khurana, George Mastorakos, Sunyang Fu, Nansu Zong, Jungwei Fan, Hongfang Liu, and Ming Huang. 2023. Classification of Patient Portal Messages with BERT-based Language Models. In 2023 IEEE 11th International Conference on Healthcare Informatics (ICHI), pages 176–182. [find](https://scholar.google.com/scholar?q=Yang+Ren%2C+Dezhi+Wu%2C+Aditya+Khurana%2C+George+Mastorakos%2C+Sunyang+Fu%2C+Nansu+Zong%2C+Jungwei+Fan%2C+Hongfang+Liu%2C+and+Ming+Huang%2E+2023%2E+Classification+of+Patient+Portal+Messages+with+BERT%2Dbased+Language+Model)
17. Sonish Sivarajkumar, Haneef Ahamed Mohammad, David Oniani, Kirk Roberts, William Hersh, Hongfang Liu, Daqing He, Shyam Visweswaran, and Yanshan Wang. 2024. Clinical Information Retrieval: A Literature Review. Journal of Healthcare Informatics Research, 8(2):313–352. [find](https://scholar.google.com/scholar?q=Sonish+Sivarajkumar%2C+Haneef+Ahamed+Mohammad%2C+David+Oniani%2C+Kirk+Roberts%2C+William+Hersh%2C+Hongfang+Liu%2C+Daqing+He%2C+Shyam+Visweswaran%2C+and+Yanshan+Wang%2E+2024%2E+Clinical+Information+Retrieval%3A+A+Literature)
18. Sarvesh Soni and Dina Demner-Fushman. 2026a. A Dataset for Addressing Patient’s Information Needs related to Clinical Course of Hospitalization. Scientific Data, 13(1):523. [find](https://scholar.google.com/scholar?q=Sarvesh+Soni+and+Dina+Demner%2DFushman%2E+2026a%2E+A+Dataset+for+Addressing+Patient%E2%80%99s+Information+Needs+related+to+Clinical+Course+of+Hospitalization%2E+Scientific+Data%2C+13%281%29%3A523%2E)
19. Sarvesh Soni and Dina Demner-Fushman. 2026b. Overview of the ArchEHR-QA 2026 shared task on grounded question answering from electronic health records. In Proceedings of the Third Workshop on Patient-Oriented Language Processing (Cl4health), Palma, Mallorca (Spain). ELRA. [find](https://scholar.google.com/scholar?q=Sarvesh+Soni+and+Dina+Demner%2DFushman%2E+2026b%2E+Overview+of+the+ArchEHR%2DQA+2026+shared+task+on+grounded+question+answering+from+electronic+health+records%2E+In+Proceedings+of+the+Third+Workshop+on+Patient%2D)
20. Ahmad P. Tafti, Sunyang Fu, Aditya Khurana, George M. Mastorakos, Kenneth G. Poole, Stephen J. Traub, James A. Yiannias, and Hongfang Liu. 2019. Artificial intelligence to organize patient portal messages: A journey from an ensemble deep learning text classification to rule-based named entity recogn… [find](https://scholar.google.com/scholar?q=Ahmad+P%2E+Tafti%2C+Sunyang+Fu%2C+Aditya+Khurana%2C+George+M%2E+Mastorakos%2C+Kenneth+G%2E+Poole%2C+Stephen+J%2E+Traub%2C+James+A%2E+Yiannias%2C+and+Hongfang+Liu%2E+2019%2E+Artificial+intelligence+to+organize+patient+portal+mess)
21. Ming Tai-Seale, Cliff W. Olson, Jinnan Li, Albert S. Chan, Criss Morikawa, Meg Durbin, Wei Wang, and Harold S. Luft. 2017. Electronic Health Record Logs Indicate That Physicians Split Time Evenly Between Seeing Patients And Desktop Medicine. Health Affairs, 36(4):655–662. [find](https://scholar.google.com/scholar?q=Ming+Tai%2DSeale%2C+Cliff+W%2E+Olson%2C+Jinnan+Li%2C+Albert+S%2E+Chan%2C+Criss+Morikawa%2C+Meg+Durbin%2C+Wei+Wang%2C+and+Harold+S%2E+Luft%2E+2017%2E+Electronic+Health+Record+Logs+Indicate+That+Physicians+Split+Time+Evenly+Betw)
22. Shweta Yadav, Deepak Gupta, Asma Ben Abacha, and Dina Demner-Fushman. 2022. Question-aware transformer models for consumer health question summarization. Journal of Biomedical Informatics, 128:104040. [find](https://scholar.google.com/scholar?q=Shweta+Yadav%2C+Deepak+Gupta%2C+Asma+Ben+Abacha%2C+and+Dina+Demner%2DFushman%2E+2022%2E+Question%2Daware+transformer+models+for+consumer+health+question+summarization%2E+Journal+of+Biomedical+Informatics%2C+128%3A104040%2E)
23. Qi Yan, Zheng Jiang, Zachary Harbin, Preston H Tolbert, and Mark G Davies. 2021. Exploring the relationship between electronic health records and provider burnout: A systematic review. Journal of the American Medical Informatics Association, 28(5):1009– 1021. [find](https://scholar.google.com/scholar?q=Qi+Yan%2C+Zheng+Jiang%2C+Zachary+Harbin%2C+Preston+H+Tolbert%2C+and+Mark+G+Davies%2E+2021%2E+Exploring+the+relationship+between+electronic+health+records+and+provider+burnout%3A+A+systematic+review%2E+Journal+of+the+)
24. Wen-wai Yim, Yujuan Fu, Asma Ben Abacha, Neal Snider, Thomas Lin, and Meliha Yetisgen. 2023. Aci-bench: A Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation. Scientific Data, 10(1):586. [find](https://scholar.google.com/scholar?q=Wen%2Dwai+Yim%2C+Yujuan+Fu%2C+Asma+Ben+Abacha%2C+Neal+Snider%2C+Thomas+Lin%2C+and+Meliha+Yetisgen%2E+2023%2E+Aci%2Dbench%3A+A+Novel+Ambient+Clinical+Intelligence+Dataset+for+Benchmarking+Automatic+Visit+Note+Generation%2E+)
25. Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. [AlignScore: Evaluating Factual Consistency with A Unified Alignment Function](https://md.research.elamur.ai/free/en/papers/zha-2023-alignscore-evaluating-factual-consistency-unified/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328–11348, Toronto, Canada. Association…
26. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2019%2E+BERTScore%3A+Evaluating+Text+Generation+with+BERT%2E+In+International+Conference+on+Learning+Representations%2E)

*Updated 1 November 2026*