---
title: "Measuring and Modifying the Readability of English Texts with"
description: "The authors also caution that \"readability\" may not be a single construct and that the results come from… 32 references, cited by 7 papers here."
authors: ["Sean Trott", "Pamela Rivière"]
year: 2024
venue: "Proceedings of the Third Workshop on Text Simplification, Accessibility and Readability (TSAR 2024)"
source: "https://aclanthology.org/2024.tsar-1.13/"
doi: "10.18653/v1/2024.tsar-1.13"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/trott-2024-measuring-modifying-readability-english-texts/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/trott-2024-measuring-modifying-readability-english-texts/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/11/) / Measuring and Modifying the Readability of English Texts with GPT-4

# Measuring and Modifying the Readability of English Texts with GPT-4

**Authors:** [Sean Trott](https://md.research.elamur.ai/free/en/authors/sean-trott/) and [Pamela Rivière](https://md.research.elamur.ai/free/en/authors/pamela-d-riviere/) · **Year:** 2024 · **Venue:** [Proceedings of the Third Workshop on Text Simplification, Accessibility and Readability (TSAR 2024)](https://md.research.elamur.ai/free/en/venues/tsar/) ([Workshop on Text Simplification, Accessibility and Readability 2024](https://md.research.elamur.ai/free/en/venues/tsar/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.tsar-1.13/) · [PDF](https://aclanthology.org/2024.tsar-1.13.pdf) · [DOI 10.18653/v1/2024.tsar-1.13](https://doi.org/10.18653/v1/2024.tsar-1.13) · [OpenAlex](https://openalex.org/W4404781341)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/trott-2024-measuring-modifying-readability-english-texts/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether an off-the-shelf large language model can serve as a readability tool, both to judge how easy an English passage is and to rewrite it toward an easier or harder level. In the first study, GPT-4 Turbo and GPT-4o mini scored all 4,724 passages of the CLEAR corpus on a 1–100 ease scale. They received only a teacher-style system prompt and a definition of readability, with no labeled examples. Their scores tracked the teacher-derived human index closely and beat both classic readability formulas and word-level psycholinguistic norms. In the second, pre-registered study, Turbo rewrote 100 CLEAR excerpts in an easier and a harder direction. Crowdworkers on Prolific then rated original and modified versions. Easier rewrites were judged easier than harder ones, although much of the variation in human ratings stayed unexplained. The work matters because it gives a cheap, reproducible baseline for LLM-based readability assessment and simplification. The authors also caution that "readability" may not be a single construct and that the results come from one dataset only.

## Key points
- **Data.** The CLEAR (CommonLit Ease of Readability) corpus contains 4,724 excerpts of 140–200 words each, covering fiction and non-fiction from 1875 to 2020. Its readability index comes from teachers' pairwise comparisons.
- **Zero-shot correlations.** The models were run at temperature 0 with a 3-token output cap. GPT-4 Turbo reached r = 0.76 with human scores and GPT-4o mini reached r = 0.74. For comparison, two random halves of the human data correlate at only r = 0.63, and the two models agree with each other at r = 0.81.
- **Psycholinguistic baseline.** A linear model on passage-averaged word frequency, concreteness and age of acquisition explained R² = 0.36 of the variance, versus about 0.58 for Turbo. In a random forest combining all features, Turbo's ratings were the most important predictor, followed by age of acquisition.
- **Study 2 design.** 100 excerpts × 3 versions (original, easier, harder) gave 300 items. These were split into 6 Latin-square lists, so no rater saw two versions of the same passage. Rewrites were limited to roughly the original length.
- **Study 2 participants and outcome.** Of 69 recruited participants, 10 were excluded and 59 remained, who rated items on a 1–5 scale. The outcome was that "easier" rewrites were rated easier than "harder" ones.

## Interesting ideas
- The authors added a "make it harder" condition purely as a control, which shows the rewriter is truly steerable rather than simply rated easier regardless of instruction.
- Raters were dropped if their scores on unmodified originals barely correlated with the gold index, so known items served as a built-in quality filter.
- Model scores were set against the split-half agreement of the human ratings themselves, which reveals how noisy the target is and frames what counts as a good score.
- Cheap lexical norms (frequency, concreteness, age of acquisition) averaged over a passage already explained a large share of difficulty, with age of acquisition the strongest interpretable cue.
- A random forest's feature importances were used to judge whether an opaque LLM score adds information beyond transparent features.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (back): when testing generation from the meaning graph at a target register, also request the opposite register and check that the measured difficulty moves both ways.
- **I2** (learn): grade LLM teacher tiers or witness parsers on a gold-anchored subset first, and exclude from silver-data voting any source that falls below a correlation threshold.
- **I3** (learn): report inter-annotator or treebank-internal agreement next to each metric, so parser or QA gains are read against a realistic ceiling.
- **I4** (tokens): store these norms as lemma attributes in crate en, giving the later reader stages an explainable difficulty signal that needs no black-box model.
- **I5** (answer): apply the same check when deciding whether an LLM-derived signal deserves a place next to the explainable features used to grade school-course reading tests.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of California San Diego](https://md.research.elamur.ai/free/en/universities/university-of-california-san-diego/) — University, San Diego, United States · 89 works · 54,032 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,379 works

## Cite

Cite as: Sean Trott and Pamela Rivière. 2024. Measuring and Modifying the Readability of English Texts with GPT-4. Proceedings of the Third Workshop on Text Simplification, Accessibility and Readability (TSAR 2024).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{trott2024measuring,
  title = {{Measuring and Modifying the Readability of English Texts with GPT-4}},
  author = {Sean Trott and Pamela Rivière},
  booktitle = {Proceedings of the Third Workshop on Text Simplification, Accessibility and Readability (TSAR 2024)},
  year = {2024},
  doi = {10.18653/v1/2024.tsar-1.13},
  url = {https://doi.org/10.18653/v1/2024.tsar-1.13}
}
```

- [← previous John vs. Ahmed: Debate-Induced Bias in Multilingual LLMs](https://md.research.elamur.ai/free/en/papers/demidova-2024-john-vs-ahmed-debate-induced/)
- [next → MediTOD: An English Dialogue Dataset for Medical History Taking with Comprehensive Annotations](https://md.research.elamur.ai/free/en/papers/saley-2024-meditod-english-dialogue-dataset-medical/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/11/)

## References

32 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alexander Anwyl-Irvine, Jessica Massonnié, Adam Flitton, Natasha Kirkham, and Jo Evershed. 2018. Gorillas in our midst: Gorilla. sc. Behavior Research Methods. [find](https://scholar.google.com/scholar?q=Alexander+Anwyl%2DIrvine%2C+Jessica+Massonni%C3%A9%2C+Adam+Flitton%2C+Natasha+Kirkham%2C+and+Jo+Evershed%2E+2018%2E+Gorillas+in+our+midst%3A+Gorilla%2E+sc%2E+Behavior+Research+Methods%2E)
2. Dale J Barr, Roger Levy, Christoph Scheepers, and Harry J Tily. 2013. Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of memory and language, 68(3):255–278. [↗](https://doi.org/10.1016/j.jml.2012.11.001)
3. Chao-Yi Chen, Jen-Hao Yang, and Lung-Hao Lee. 2023. NCUEE-NLP at BioLaySumm task 2: Readability-controlled summarization of biomedical articles using the PRIMERA models. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 586–591, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chao%2DYi+Chen%2C+Jen%2DHao+Yang%2C+and+Lung%2DHao+Lee%2E+2023%2E+NCUEE%2DNLP+at+BioLaySumm+task+2%3A+Readability%2Dcontrolled+summarization+of+biomedical+articles+using+the+PRIMERA+models%2E+In+The+22nd+Workshop+on+Biomed)
4. Mathias Creutz. 2024. Correcting challenging Finnish learner texts with claude, GPT-3.5 and GPT-4 large language models. In Proceedings of the Ninth Workshop on Noisy and User-generated Text (W-NUT 2024), pages 1–10, San Ġiljan, Malta. Association for Computational Linguistics. [↗](https://doi.org/10.48448/jz7y-6e77)
5. Ziggy Cross, Michelle Yun, Ananya Apparaju, Jata Mac-Cabe, Garrett Nicolai, and Miikka Silfverberg. 2023. Glossy bytes: Neural glossing using subword encoding. In Proceedings of the 20th SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 222–229, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ziggy+Cross%2C+Michelle+Yun%2C+Ananya+Apparaju%2C+Jata+Mac%2DCabe%2C+Garrett+Nicolai%2C+and+Miikka+Silfverberg%2E+2023%2E+Glossy+bytes%3A+Neural+glossing+using+subword+encoding%2E+In+Proceedings+of+the+20th+SIGMORPHON+wo)
6. Scott Crossley, Joon Suh Choi, Yanisa Scherber, and Mathis Lucka. 2023a. Using large language models to develop readability formulas for educational settings. In International Conference on Artificial Intelligence in Education, pages 422–427. Springer. [find](https://scholar.google.com/scholar?q=Scott+Crossley%2C+Joon+Suh+Choi%2C+Yanisa+Scherber%2C+and+Mathis+Lucka%2E+2023a%2E+Using+large+language+models+to+develop+readability+formulas+for+educational+settings%2E+In+International+Conference+on+Artificial)
7. Scott Crossley, Aron Heintz, Joon Suh Choi, Jordan Batchelor, Mehrnoush Karimi, and Agnes Malatinszky. 2023b. A large-scaled corpus for assessing text readability. Behavior Research Methods, 55(2):491– 507. [↗](https://doi.org/10.3758/s13428-022-01802-x)
8. Douglas Bates. 2011. Mixed models in r using the lme4 package part 5: Generalized linear mixed models. University of Wisconsin: Madison, WI, USA. [find](https://scholar.google.com/scholar?q=Douglas+Bates%2E+2011%2E+Mixed+models+in+r+using+the+lme4+package+part+5%3A+Generalized+linear+mixed+models%2E+University+of+Wisconsin%3A+Madison%2C+WI%2C+USA%2E)
9. Scott A Crossley, Jerry Greenfield, and Danielle S Mc-Namara. 2008. Assessing text readability using cognitively based indices. Tesol Quarterly, 42(3):475– 493. [↗](https://doi.org/10.1002/j.1545-7249.2008.tb00142.x)
10. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
11. Tovly Deutsch, Masoud Jasbi, and Stuart Shieber. 2020. [Linguistic features for readability assessment](https://md.research.elamur.ai/free/en/papers/deutsch-2020-linguistic-features-readability-assessment/). In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 1–17, Seattle, WA, USA → Online. Association for Computational Linguistics.
12. Asma Farajidizaji, Vatsal Raina, and Mark Gales. 2023. Is it possible to modify text to a target readability level? an initial investigation using zero-shot large language models. arXiv preprint arXiv:2309.12551. [↗](https://arxiv.org/abs/2309.12551)
13. Rudolph Flesch. 1948. A new readability yardstick. Journal of applied psychology, 32(3):221. [↗](https://doi.org/10.1037/h0057532)
14. Sohom Ghosh, Shovon Sengupta, Sudip Naskar, and Sunny Kumar Singh. 2022. FinRAD: Financial readability assessment dataset - 13,000+ definitions of financial terms for measuring readability. In Proceedings of the 4th Financial Narrative Processing Workshop @LREC2022, pages 1–9, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Sohom+Ghosh%2C+Shovon+Sengupta%2C+Sudip+Naskar%2C+and+Sunny+Kumar+Singh%2E+2022%2E+FinRAD%3A+Financial+readability+assessment+dataset+%2D+13%2C000%2B+definitions+of+financial+terms+for+measuring+readability%2E+In+Proceed)
15. Joseph Marvin Imperial and Harish Tayyar Madabushi. 2023. Flesch or fumble? evaluating readability standard alignment of instruction-tuned language models. In Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 205–223, Singapore. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2309.05454)
16. Tannon Kew, Alison Chi, Laura Vásquez-Rodríguez, Sweta Agrawal, Dennis Aumiller, Fernando Alva-Manchego, and Matthew Shardlow. 2023. BLESS: Benchmarking large language models on sentence simplification. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13291–13309, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tannon+Kew%2C+Alison+Chi%2C+Laura+V%C3%A1squez%2DRodr%C3%ADguez%2C+Sweta+Agrawal%2C+Dennis+Aumiller%2C+Fernando+Alva%2DManchego%2C+and+Matthew+Shardlow%2E+2023%2E+BLESS%3A+Benchmarking+large+language+models+on+sentence+simplificatio)
17. Victor Kuperman, Hans Stadthagen-Gonzalez, and Marc Brysbaert. 2012. Age-of-acquisition ratings for 30,000 english words. Behavior research methods, 44:978–990. [↗](https://doi.org/10.3758/s13428-012-0210-4)
18. Kristopher Kyle, Scott Crossley, and Cynthia Berger. 2018. The tool for the automatic analysis of lexical sophistication (taales): version 2.0. Behavior research methods, 50:1030–1046. [↗](https://doi.org/10.3758/s13428-017-0924-4)
19. Kristopher Kyle and Scott A Crossley. 2015. Automatically assessing lexical sophistication: Indices, tools, findings, and application. Tesol Quarterly, 49(4):757– 786. [↗](https://doi.org/10.1002/tesq.194)
20. Zihao Li, Samuel Belkadi, Nicolo Micheletti, Lifeng Han, Matthew Shardlow, and Goran Nenadic. 2023. Large language models and control mechanisms improve text readability of biomedical abstracts. arXiv preprint arXiv:2309.13202. [↗](https://arxiv.org/abs/2309.13202)
21. Dragomir Radev, Chien-Sheng Wu, and Arman Cohan. 2023. Benchmarking generation and evaluation capabilities of large language models for instruction controllable summarization. arXiv preprint arXiv:2311.09184. [↗](https://arxiv.org/abs/2311.09184)
22. Bertha A Lively and SL Pressey. 1923. A method for measuring the” vocabulary burden” of textbooks: Educational administration and supervision,”. A method for measuring the” vocabulary burden” of textbooks: Educational Administration and Supervision. [find](https://scholar.google.com/scholar?q=Bertha+A+Lively+and+SL+Pressey%2E+1923%2E+A+method+for+measuring+the%E2%80%9D+vocabulary+burden%E2%80%9D+of+textbooks%3A+Educational+administration+and+supervision%2C%E2%80%9D%2E+A+method+for+measuring+the%E2%80%9D+vocabulary+burden%E2%80%9D+of+textb)
23. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2022. Readability controllable biomedical document summarization. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4667–4680, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2022.findings-emnlp.343)
24. Matej Martinc, Senja Pollak, and Marko Robnik-Šikonja. 2021. [Supervised and unsupervised neural approaches to text readability](https://md.research.elamur.ai/free/en/papers/martinc-2021-supervised-unsupervised-neural-approaches-text/). Computational Linguistics, 47(1):141–179. [↗](https://doi.org/10.1162/coli_a_00398)
25. Hiroki Nomoto. 2023. Issues surrounding the use of ChatGPT in similar languages: The case of Malay and Indonesian. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 76–82, Nusa Dua, Bali. Association for Computational Linguistics. [↗](https://doi.org/10.48448/52dv-8502)
26. Gustavo Paetzold and Lucia Specia. 2017. Lexical simplification with neural ranking. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 34–40, Valencia, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Gustavo+Paetzold+and+Lucia+Specia%2E+2017%2E+Lexical+simplification+with+neural+ranking%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Linguistics%3A+Vol)
27. Nirmal Patel, Pooja Nagpal, Tirth Shah, Aditya Sharma, Shrey Malvi, and Derek Lomas. 2023. Improving mathematics assessment readability: Do large language models help? Journal of Computer Assisted Learning, 39(3):804–822. [↗](https://doi.org/10.1111/jcal.12776)
28. Dongqi Pu and Vera Demberg. 2023. [ChatGPT vs human-authored text: Insights into controllable text summarization and sentence style transfer](https://md.research.elamur.ai/free/en/papers/liu-2023-chatgpt-vs-human-authored-text/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 1–18, Toronto, Canada. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2023.acl-srw.1)
29. Fengkai Liu and John Lee. 2023. Hybrid models for sentence readability assessment. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 448– 454, Toronto, Canada. Association for Computational Linguistics. [↗](https://doi.org/10.48448/x21q-5z65)
30. Yu Qiao, Xiaofei Li, Daniel Wiechmann, and Elma Kerz. 2022. (psycho-)linguistic features meet transformer models for improved explainable and controllable text simplification. In Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022), pages 125–146, Abu Dhabi, United Arab Emirates (Virtual). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yu+Qiao%2C+Xiaofei+Li%2C+Daniel+Wiechmann%2C+and+Elma+Kerz%2E+2022%2E+%28psycho%2D%29linguistic+features+meet+transformer+models+for+improved+explainable+and+controllable+text+simplification%2E+In+Proceedings+of+the+Wo)
31. Yixin Liu, Alexander R Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Inioluwa Deborah Raji, Emily Denton, Emily M Bender, Alex Hanna, and Amandalynne Paullada. 2021. AI and the Everything in the Whole Wide World Benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). [find](https://scholar.google.com/scholar?q=Yixin+Liu%2C+Alexander+R+Fabbri%2C+Jiawen+Chen%2C+Yilun+Zhao%2C+Simeng+Han%2C+Shafiq+Joty%2C+Pengfei+Liu%2C+Inioluwa+Deborah+Raji%2C+Emily+Denton%2C+Emily+M+Bender%2C+Alex+Hanna%2C+and+Amandalynne+Paullada%2E+2021%2E+AI+and+th)
32. Leonardo F. R. Ribeiro, Mohit Bansal, and Markus Dreyer. 2023. Generating summaries with controllable readability levels. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11669–11687, Singapore. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2023.emnlp-main.714)

*Updated 1 November 2026*