---
title: "Evaluating LLM Style Transfer Through — Maro 2026"
description: "The paper asks whether an LLM can rewrite health information so that it suits readers of a given age, a key capability for conversational agents that…"
authors: ["Maria Di Maro", "Antonio Origlia", "Leonilda Bilo", "Roberta Meo", "Pietro Maturi", "Francesca Nappo"]
year: 2026
venue: "Proceedings of the 16th International Workshop on Spoken Dialogue System Technology"
source: "https://aclanthology.org/2026.iwsds-1.31/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/maro-2026-evaluating-llm-style-transfer-through/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/maro-2026-evaluating-llm-style-transfer-through/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2026](https://md.research.elamur.ai/free/papers/2026/page/2/) / Evaluating LLM Style Transfer Through Readability-Based Age Assessments

# Evaluating LLM Style Transfer Through Readability-Based Age Assessments

**Authors:** Maria Di Maro, Antonio Origlia, Leonilda Bilo, Roberta Meo, Pietro Maturi, Francesca Nappo · **Year:** 2026 · **Venue:** Proceedings of the 16th International Workshop on Spoken Dialogue System Technology · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.iwsds-1.31/) · [PDF](https://aclanthology.org/2026.iwsds-1.31.pdf)

## Summary
The paper asks whether an LLM can rewrite health information so that it suits readers of a given age, a key capability for conversational agents that disseminate medical knowledge. The work is part of the SPECIAL project, which uses conversational AI to fight stigma around epilepsy. The authors took 21 Italian scientific texts and prompted LLaMA 3.3 to produce an original popular-science version of each one for 11 target age bands, ranging from 8–9 to 18–22 years. This yielded 231 generated texts. They assessed the outputs in two ways. First, they computed the Automated Readability Index (ARI), which maps text complexity to age and grade level. Second, fifteen epileptologists judged both the perceived educational level and the factual accuracy of the texts. Both evaluations showed that complexity rose steadily with the requested age, so stylistic adaptation works. Content accuracy was usually rated medium to high, but the experts found clinically relevant mistakes. The authors therefore argue that LLMs should handle the form of a text while its content stays under tighter control.

## Key points
- **Corpus:** 21 Italian scientific epilepsy texts drawn from the SPECIAL knowledge base (sources include LICE and ILAE) were adapted by LLaMA 3.3 into 11 age groups, giving 231 texts.
- **Automatic evaluation:** ARI scores differed strongly across categories (Kruskal–Wallis, p \< 2.2e-16). Bonferroni-corrected pairwise Wilcoxon tests separated the scientific texts and the youngest groups from the higher levels.
- **Human evaluation:** 15 expert epileptologists from the LICE community completed a Qualtrics survey. Each rated four texts, one from each macro-age group (8–11, 11–14, 14–17, 17–specialist), on 1–5 scales for accuracy and minimum required education.
- **Accuracy ratings:** Mean correctness scores ranged from 3 (ages 16–17) to 4.6 (ages 14–15), and they tended to drop for older target groups, whose texts contain more technical detail.
- **Reported problems:** Experts noted the following issues:
  - The texts read as if machine-translated, including the original scientific ones.
  - Some terminology was wrong ("attacks" used instead of "seizures").
  - Some claims about breastfeeding and physical activity were misleading.
  - Seizure-management instructions were incomplete, and the role of diazepam was misdescribed.
  - The legal basis of driving restrictions was misrepresented.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{maro2026evaluating,
  title = {{Evaluating LLM Style Transfer Through Readability-Based Age Assessments}},
  author = {Maro, Maria Di and Origlia, Antonio and Bilo, Leonilda and Meo, Roberta and Maturi, Pietro and Nappo, Francesca},
  booktitle = {Proceedings of the 16th International Workshop on Spoken Dialogue System Technology},
  year = {2026},
  url = {https://aclanthology.org/2026.iwsds-1.31/}
}
```

- [← previous Evaluating Frontier LLM Translation Capability for Lakota](https://md.research.elamur.ai/free/papers/robertson-2026-evaluating-frontier-llm-translation-capability/)
- [next → Evaluating Machine Translation Datasets for Low-Web Data Languages: A Gendered Lens](https://md.research.elamur.ai/free/papers/nigatu-2026-evaluating-machine-translation-datasets-low/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/papers/2026/page/2/)

## References

22 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multi-task, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023. [↗](https://arxiv.org/abs/2302.04023) [our card](https://md.research.elamur.ai/free/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/)
2. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1–45. [↗](https://doi.org/10.1145/3641289) [our card](https://md.research.elamur.ai/free/papers/chang-2023-survey-evaluation-large-language-models/)
3. Meri Coleman and Ta Lin Liau. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2):283. [↗](https://doi.org/10.1037/h0076540)
4. Felice Dell’Orletta, Simonetta Montemagni, and Giulia Venturi. 2011. Read–it: Assessing readability of italian texts with a view to text simplification. In Proceedings of the second workshop on speech and language processing for assistive technologies, pages 73–83. [find](https://scholar.google.com/scholar?q=Felice+Dell%E2%80%99Orletta%2C+Simonetta+Montemagni%2C+and+Giulia+Venturi%2E+2011%2E+Read%E2%80%93it%3A+Assessing+readability+of+italian+texts+with+a+view+to+text+simplification%2E+In+Proceedings+of+the+second+workshop+on+speech) [our card](https://md.research.elamur.ai/free/papers/orletta-2011-read-it-assessing-readability-italian/)
5. Chenyang Lyu, Jitao Xu, and Longyue Wang. 2023. New trends in machine translation using large language models: Case examples with chatgpt. arXiv preprint arXiv:2305.01181. [↗](https://arxiv.org/abs/2305.01181)
6. G Harry Mc Laughlin. 1969. Smog grading-a new readability formula. Journal of reading, 12(8):639–646. [find](https://scholar.google.com/scholar?q=G+Harry+Mc+Laughlin%2E+1969%2E+Smog+grading%2Da+new+readability+formula%2E+Journal+of+reading%2C+12%288%29%3A639%E2%80%93646%2E)
7. Tricia A Miller. 2016. Health literacy and adherence to medical treatment in chronic and acute illness: a meta-analysis. Patient education and counseling, 99(7):1079–1086. [↗](https://doi.org/10.1016/j.pec.2016.01.020)
8. Dongqi Pu and Vera Demberg. 2023. Chatgpt vs human-authored text: Insights into controllable text summarization and sentence style transfer. arXiv preprint arXiv:2306.07799. [↗](https://arxiv.org/abs/2306.07799) [our card](https://md.research.elamur.ai/free/papers/liu-2023-chatgpt-vs-human-authored-text/)
9. Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may i introduce the gyafc dataset: Corpus, benchmarks and metrics for formality style transfer. arXiv preprint arXiv:1803.06535. [↗](https://arxiv.org/abs/1803.06535) [our card](https://md.research.elamur.ai/free/papers/rao-2018-dear-sir-or-madam-may/)
10. SC Ratzan and RM Parker. 2000. Health literacy. National library of medicine current bibliographies in medicine. Bethesda: National Institutes of Health, US Department of Health and Human Services. [find](https://scholar.google.com/scholar?q=SC+Ratzan+and+RM+Parker%2E+2000%2E+Health+literacy%2E+National+library+of+medicine+current+bibliographies+in+medicine%2E+Bethesda%3A+National+Institutes+of+Health%2C+US+Department+of+Health+and+Human+Services%2E)
11. Nicholas Riccardi, Xuan Yang, and Rutvik H Desai. 2024. The two word test as a semantic benchmark for large language models. Scientific Reports, 14(1):21593. [↗](https://doi.org/10.1038/s41598-024-72528-3)
12. Irving Rootman and Deborah Gordon-El-Bihbety. 2008. A vision for a health literate canada. Ottawa, ON: Canadian Public Health Association, page 50. [find](https://scholar.google.com/scholar?q=Irving+Rootman+and+Deborah+Gordon%2DEl%2DBihbety%2E+2008%2E+A+vision+for+a+health+literate+canada%2E+Ottawa%2C+ON%3A+Canadian+Public+Health+Association%2C+page+50%2E)
13. Rudolph Flesch. 1948. A new readability yardstick. Journal of Applied Psychology, 32(3):p221 – 233. [↗](https://doi.org/10.1037/h0057532)
14. Adem Gencer. 2024. Readability analysis of chatgpt’s responses on lung cancer. Scientific Reports, 14(1):17234. [↗](https://doi.org/10.1038/s41598-024-67293-2)
15. Robert Gunning. 1952. The technique of clear writing. (No Title). [find](https://scholar.google.com/scholar?q=Robert+Gunning%2E+1952%2E+The+technique+of+clear+writing%2E+%28No+Title%29%2E)
16. Lindsay C Kobayashi, Jane Wardle, and Christian von Wagner. 2015. Internet use, social engagement and health literacy decline during ageing in a longitudinal cohort of older english adults. J Epidemiol Community Health, 69(3):278–283. [↗](https://doi.org/10.1136/jech-2014-204733)
17. Noah Lee, Na Min An, and James Thorne. 2023. Can large language models infer and disagree like humans? CoRR. [find](https://scholar.google.com/scholar?q=Noah+Lee%2C+Na+Min+An%2C+and+James+Thorne%2E+2023%2E+Can+large+language+models+infer+and+disagree+like+humans%3F+CoRR%2E)
18. Francesco Sabatini. 1999. ‘rigidità-esplicitezza’vs ‘elasticità-implicitezza’: possibili parametri massimi per una tipologia dei testi. Linguistica testuale comparativa, pages 142–172. [find](https://scholar.google.com/scholar?q=Francesco+Sabatini%2E+1999%2E+%E2%80%98rigidit%C3%A0%2Desplicitezza%E2%80%99vs+%E2%80%98elasticit%C3%A0%2Dimplicitezza%E2%80%99%3A+possibili+parametri+massimi+per+una+tipologia+dei+testi%2E+Linguistica+testuale+comparativa%2C+pages+142%E2%80%93172%2E)
19. Georgelle Thomas, R Derald Hartley, and J Peter Kincaid. 1975. Test-retest and inter-analyst reliability of the automated readability index, flesch reading ease score, and the fog count. Journal of Reading Behavior, 7(2):149–154. [↗](https://doi.org/10.1080/10862967509547131)
20. Kai-Cheng Yang and Filippo Menczer. 2025. Accuracy and political bias of news source credibility ratings by large language models. In Proceedings of the 17th ACM Web Science Conference 2025, pages 127–137. [find](https://scholar.google.com/scholar?q=Kai%2DCheng+Yang+and+Filippo+Menczer%2E+2025%2E+Accuracy+and+political+bias+of+news+source+credibility+ratings+by+large+language+models%2E+In+Proceedings+of+the+17th+ACM+Web+Science+Conference+2025%2C+pages+127)
21. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414. [↗](https://arxiv.org/abs/2210.02414) [our card](https://md.research.elamur.ai/free/papers/zeng-2022-glm-130b-open-bilingual-pre/)
22. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. [↗](https://arxiv.org/abs/2211.09110)