---
title: "Linguistic Features for Readability Assessment — Deutsch 2020"
description: "The paper asks whether hand-crafted linguistic features still help readability assessment now that… 16 references, cited by 64 papers here."
authors: ["Tovly Deutsch", "Masoud Jasbi", "Stuart Shieber"]
year: 2020
venue: "Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications"
source: "https://aclanthology.org/2020.bea-1.1/"
doi: "10.18653/v1/2020.bea-1.1"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/deutsch-2020-linguistic-features-readability-assessment/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/deutsch-2020-linguistic-features-readability-assessment/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) / Linguistic Features for Readability Assessment

# Linguistic Features for Readability Assessment

**Authors:** Tovly Deutsch, Masoud Jasbi and [Stuart Shieber](https://md.research.elamur.ai/free/en/authors/stuart-m-shieber/) · **Year:** 2020 · **Venue:** [Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications](https://md.research.elamur.ai/free/en/venues/bea/) ([BEA 2020](https://md.research.elamur.ai/free/en/venues/bea/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.bea-1.1/) · [PDF](https://aclanthology.org/2020.bea-1.1.pdf) · [DOI 10.18653/v1/2020.bea-1.1](https://doi.org/10.18653/v1/2020.bea-1.1) · [OpenAlex](https://openalex.org/W3037554857)

**Citations:** 66 citations (OpenAlex, 2026-10-14) · [64](https://md.research.elamur.ai/free/en/papers/deutsch-2020-linguistic-features-readability-assessment/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks whether hand-crafted linguistic features still help readability assessment now that deep neural models lead on this task. The authors combine the two approaches. Predictions from deep models (a transformer, a hierarchical attention network (HAN) and CNNs) become input features, which are concatenated with linguistic measures and passed to a downstream classifier such as an SVM. The linguistic measures include an established set based on Vajjala Balakrishna (2015) plus new features targeting syntactic ambiguity and syntactic diversity. Experiments use two of the largest readability corpora, WeeBit and Newsela, and also vary the amount of training data.

The main finding is negative. Once enough training data is available, adding linguistic features does not raise state-of-the-art results. The features only help noticeably when training sets are tiny. The authors take this as early evidence that deep models already encode much of the readability-relevant linguistic information. They suggest that collecting more labelled text is usually a better investment than engineering features. This matters for anyone choosing between feature engineering and data collection for low-resource text-difficulty tasks.

## Key points
- **Corpora:**
  - WeeBit combines WeeklyReader and BBC Bitesize, targets readers aged 7–16, and has roughly 6,000 documents. A downsampled version keeps 625 documents per class.
  - Newsela has 1,911 news articles, each rewritten up to 4 times at different grade levels (grades 2–12). This weakens the usual link between topic and difficulty.
- **Existing features:** The feature set comes from software by Vajjala Balakrishna (2015), covering 86 features from prior literature. It includes:
  - traditional formula features such as Flesch-Kincaid;
  - POS and phrase counts from the Stanford parser;
  - word type proportions;
  - clause and t-unit ratios from Lu (2010);
  - psycholinguistic resources (Celex, MRC).
- **Baselines:** The deep models follow Martinc et al. (2019), where a pretrained transformer was best on WeeBit and a HAN was best on Newsela.
- **Training-size experiment:** CNN models with and without linguistic features were trained on subsets of downsampled WeeBit, scored with macro F1 on a fixed test set. The features helped only below about 200 training documents.
- **Code:** The code is public at github.com/TovlyDeutsch/Linguistic-Features-for-Readability.

## Interesting ideas
- **Stacking for fusion:** Neural model outputs are treated as ordinary features and stacked with hand-crafted measures in a simple classifier. This is a cheap way to test whether expert features add anything beyond a neural model, with no changes to the network architecture.
- **Three label encodings for ordinal levels:** - plain classes; - regression on the midpoint of the target age range; - regression on evenly spaced class ranks.  The rank-based option deliberately assumes equal difficulty steps, since a one-year age gap may matter more for young readers than for teenagers. This comparison is reusable for any graded scale, such as proficiency levels.
- **Learning curve with a fixed test set:** Training size is varied while the test set stays the same, so the effect of data quantity is isolated. This shows directly at what data size an inductive bias stops being worth the effort, which supports budgeting decisions between annotation and feature work.
- **Corpora that separate topic from level:** Newsela's design, with the same article rewritten at several levels, keeps a model from inferring difficulty from subject matter. This is a useful principle when building any new graded-text dataset.
- **Interpreting a null result:** The authors treat the lack of gains as a hypothesis that neural models already represent these features. They propose probing studies to test it, and they flag inconsistent corpus labelling as a possible ceiling on performance, suggesting averaging over multiple annotators.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Harvard University](https://md.research.elamur.ai/free/en/universities/harvard-university/) — University, Cambridge, United States · 73 works · 37,309 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Tovly Deutsch, Masoud Jasbi and Stuart Shieber. 2020. Linguistic Features for Readability Assessment. Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{deutsch2020linguistic,
  title = {{Linguistic Features for Readability Assessment}},
  author = {Tovly Deutsch, Masoud Jasbi and Stuart Shieber},
  booktitle = {Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications},
  year = {2020},
  doi = {10.18653/v1/2020.bea-1.1},
  url = {https://doi.org/10.18653/v1/2020.bea-1.1}
}
```

- [← previous It's Morphin' Time! Combating Linguistic Discrimination with Inflectional Perturbations](https://md.research.elamur.ai/free/en/papers/tan-2020-it-s-morphin-time-combating/)
- [next → Mapping Natural Language Instructions to Mobile UI Action Sequences](https://md.research.elamur.ai/free/en/papers/li-2020-mapping-natural-language-instructions-mobile/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/)

## References

16 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [Martı́n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenbe…](https://md.research.elamur.ai/free/en/papers/klein-2003-accurate-unlexicalized-parsing/)
2. Solomon Kullback and Richard A. Leibler. 1951. On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1):79–86. [find](https://scholar.google.com/scholar?q=Solomon+Kullback+and+Richard+A%2E+Leibler%2E+1951%2E+On+Information+and+Sufficiency%2E+The+Annals+of+Mathematical+Statistics%2C+22%281%29%3A79%E2%80%9386%2E)
3. Menglin Xia, Ekaterina Kochmar, and Ted Briscoe. 2016. [Text Readability Assessment for Second Language Learners](https://md.research.elamur.ai/free/en/papers/xia-2016-text-readability-assessment-second-language/). In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, pages 12–22, San Diego, CA. Association for Computational Linguistics.
4. Victor Kuperman, Hans Stadthagen-Gonzalez, and Marc Brysbaert. 2012. Age-of-acquisition ratings for 30,000 English words. Behavior Research Methods, 44(4):978–990. [find](https://scholar.google.com/scholar?q=Victor+Kuperman%2C+Hans+Stadthagen%2DGonzalez%2C+and+Marc+Brysbaert%2E+2012%2E+Age%2Dof%2Dacquisition+ratings+for+30%2C000+English+words%2E+Behavior+Research+Methods%2C+44%284%29%3A978%E2%80%93990%2E)
5. Xiaofei Lu. 2010. Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics, 15(4):474–496. [find](https://scholar.google.com/scholar?q=Xiaofei+Lu%2E+2010%2E+Automatic+analysis+of+syntactic+complexity+in+second+language+writing%2E+International+Journal+of+Corpus+Linguistics%2C+15%284%29%3A474%E2%80%93496%2E)
6. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. [Hierarchical Attention Networks for Document Classification](https://md.research.elamur.ai/free/en/papers/yang-2016-hierarchical-attention-networks-document-classification/). In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 14…
7. Matej Martinc, Senja Pollak, and Marko Robnik-Šikonja. 2019. [Supervised and unsupervised neural approaches to text readability](https://md.research.elamur.ai/free/en/papers/martinc-2021-supervised-unsupervised-neural-approaches-text/). Computing Research Repository, arXiv:1503.06733. Version 2.
8. Viet Nguyen. 2020. Hierarchical Attention Networks for Document Classification. Original-date: 2019-01-31T18:56:40Z. [find](https://scholar.google.com/scholar?q=Viet+Nguyen%2E+2020%2E+Hierarchical+Attention+Networks+for+Document+Classification%2E+Original%2Ddate%3A+2019%2D01%2D31T18%3A56%3A40Z%2E)
9. Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/). Computing Research Repository, arXiv:1506.06724.
10. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn… [find](https://scholar.google.com/scholar?q=Fabian+Pedregosa%2C+Gae%CC%88l+Varoquaux%2C+Alexandre+Gramfort%2C+Vincent+Michel%2C+Bertrand+Thirion%2C+Olivier+Grisel%2C+Mathieu+Blondel%2C+Peter+Prettenhofer%2C+Ron+Weiss%2C+Vincent+Dubourg%2C+Jake+Vanderplas%2C+Alexandre+Pas)
11. Sarah E. Schwarm and Mari Ostendorf. 2005. [Reading Level Assessment Using Support Vector Machines and Statistical Language Models](https://md.research.elamur.ai/free/en/papers/schwarm-2005-reading-level-assessment-support-vector/). In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL ’05, pages 523–530, Stroudsburg, PA, USA. Association for Computational Lingu…
12. Sowmya Vajjala and Detmar Meurers. 2012. [On Improving the Accuracy of Readability Classification using Insights from Second Language Acquisition](https://md.research.elamur.ai/free/en/papers/vajjala-2012-improving-accuracy-readability-classification-insights/). In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 163– 173, Montréal, Canada. Association for Computational Li…
13. Sowmya Vajjala Balakrishna. 2015. Analyzing Text Complexity and Text Simplification: Connecting Linguistics, Processing and Educational Applications. Dissertation, Universität Tübingen. [find](https://scholar.google.com/scholar?q=Sowmya+Vajjala+Balakrishna%2E+2015%2E+Analyzing+Text+Complexity+and+Text+Simplification%3A+Connecting+Linguistics%2C+Processing+and+Educational+Applications%2E+Dissertation%2C+Universita%CC%88t+Tu%CC%88bingen%2E)
14. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in neural information processing systems 30, pages 5998–6008. Curran Associates, Inc.
15. Michael Wilson. 1988. MRC psycholinguistic database: Machine-usable dictionary, version 2.00. Behavior Research Methods, Instruments, & Computers, 20(1):6–10. [find](https://scholar.google.com/scholar?q=Michael+Wilson%2E+1988%2E+MRC+psycholinguistic+database%3A+Machine%2Dusable+dictionary%2C+version+2%2E00%2E+Behavior+Research+Methods%2C+Instruments%2C+%26+Computers%2C+20%281%29%3A6%E2%80%9310%2E)
16. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace’s [transformers: state-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). Technical report.

*Updated 24 October 2026*