---
title: "Reading Level Assessment Using Support Vector — Schwarm 2005"
description: "The paper addresses automatic reading level estimation."
authors: ["Sarah Schwarm", "Mari Ostendorf"]
year: 2005
venue: "Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05)"
source: "https://aclanthology.org/P05-1065/"
doi: "10.3115/1219840.1219905"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/schwarm-2005-reading-level-assessment-support-vector/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/schwarm-2005-reading-level-assessment-support-vector/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2005](https://md.research.elamur.ai/free/en/papers/2005/) / Reading Level Assessment Using Support Vector Machines and Statistical Language Models

# Reading Level Assessment Using Support Vector Machines and Statistical Language Models

**Authors:** Sarah Schwarm, [Mari Ostendorf](https://md.research.elamur.ai/free/en/authors/mari-ostendorf/) · **Year:** 2005 · **Venue:** Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P05-1065/) · [PDF](https://aclanthology.org/P05-1065.pdf) · [DOI 10.3115/1219840.1219905](https://doi.org/10.3115/1219840.1219905) · [OpenAlex](https://openalex.org/W2153081451)

**Citations:** 318 citations (openalex-snapshot · 2026-10-14) · [129](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper addresses automatic reading level estimation. The motivating use case is helping bilingual-education teachers find topical texts that are easy enough for English learners. Classic formulas such as Flesch-Kincaid and Lexile rely on sentence length, syllable counts or fixed word lists, so they miss real syntactic difficulty and penalise necessary topic vocabulary.

The authors train on the Weekly Reader corpus, which has roughly 2,400 articles from grade 2–5 editions. They add two supplemental corpora of adult texts paired with simplified versions: Encyclopedia Britannica/Britannica Elementary and CNN/abridged CNN.

They build one detector per grade. Each detector uses n-gram language model perplexities as features, alongside parse-based features from the Charniak parser and traditional measures. An SVM combines all of these. Trigram models outperform bigram and unigram models, and the SVM combination gives the best results overall.

Compared with Flesch-Kincaid and Lexile, the SVM misclassifies far fewer articles by more than one grade. It also behaves sensibly on out-of-domain Washington Post texts. This shows that cheap, corpus-trainable LM features capture readability better than static formulas.

## Key points
- **Data:**
  - Weekly Reader grades 2/3/4/5 contain 351/589/766/691 articles respectively.
  - Supplemental corpora: Britannica (115 articles) with Britannica Elementary (115), and CNN (111) with CNN Abridged (111).
- **LM setup:**
  - Models were trained with the SRI Language Modeling Toolkit using modified Kneser-Ney smoothing.
  - Vocabulary was selected by information gain.
  - Perplexity worked better than the likelihood ratio as an SVM feature.
- **Detection performance:** Equal error rates on the DET curves range from about 12% to 25% across grades. At the chosen operating points, test-set precision/recall were:

  | Grade | Precision | Recall |
  |---|---|---|
  | 2 | 38% | 61% |
  | 3 | 38% | 87% |
  | 4 | 70% | 60% |
  | 5 | 75% | 79% |

- **Comparison with traditional measures:** Share of articles misclassified by more than one grade:

  | Grade | SVM | Lexile | Flesch-Kincaid |
  |---|---|---|---|
  | 2 | 5.5% | 33% | 78% |
  | 3 | 3.3% | 27% | 67% |
  | 4 | 13% | 26% | 74% |
  | 5 | 21% | 24% | 59% |

- **Out-of-domain check:**
  - Of 10 "Kidspost" articles (aimed at grades 3–8), the classifier labelled four as grade 4 and seven as grade 5, with one article receiving both labels.
  - Adult Washington Post articles were assigned the highest grade available.

## Interesting ideas
- **One yes/no detector per level instead of a single multi-class ranker.** This matches how a teacher actually searches ("give me grade-3 texts"). Each detector's threshold can be tuned independently, which suits retrieval-style applications.
- **Language model perplexities as features.** Training a separate n-gram model per level and feeding its perplexities into a discriminative classifier gives a cheap stand-in for both lexical and syntactic difficulty. This transfers easily to languages or domains without readability formulas, such as Ukrainian.
- **Parallel original/simplified corpora as extra training data.** Pairs like Britannica vs. Britannica Elementary, or CNN vs. abridged CNN, can train coarse "easy vs. hard" models. Their outputs then serve as features for finer classification, a useful trick when fine-grained labels are scarce.
- **An error metric that reflects ordered labels.** Counting predictions that are off by more than one level penalises large ordinal mistakes. It is more informative than plain accuracy for graded scales.
- **Operating points chosen on a development set via DET curves.** Picking thresholds this way, and reporting precision/recall at those points, makes the tradeoff explicit. It shows clearly why middle levels (grades 3–4) are harder than the extremes.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

129 papers of our library cite it (of the 140884 whose bibliographies we read); 13 of them have a page here, 116 not yet.

- [Modélisation de la lisibilité en français pour les personnes en situation d'illettrisme](https://md.research.elamur.ai/free/en/papers/aissa-2025-modelisation-de-la-lisibilite-en/)
- [Readability Assessment for Text Simplification](https://md.research.elamur.ai/free/en/papers/aluisio-2010-readability-assessment-text-simplification/)
- [Modeling Local Coherence: An Entity-Based Approach](https://md.research.elamur.ai/free/en/papers/barzilay-2008-modeling-local-coherence-entity-based/)
- [A Comparison of Features for Automatic Readability Assessment](https://md.research.elamur.ai/free/en/papers/feng-2010-comparison-features-automatic-readability-assessment/)
- [An “AI readability” Formula for French as a Foreign Language](https://md.research.elamur.ai/free/en/papers/francois-2012-ai-readability-formula-french-as/)
- [GDEX: Automatically Finding Good Dictionary Examples in a Corpus](https://md.research.elamur.ai/free/en/papers/lex-2008-kilgarriff-gdex-examples/)
- [Supervised and Unsupervised Neural Approaches to Text Readability](https://md.research.elamur.ai/free/en/papers/martinc-2021-supervised-unsupervised-neural-approaches-text/)
- [READ–IT: Assessing Readability of Italian Texts with a View to Text Simplification](https://md.research.elamur.ai/free/en/papers/orletta-2011-read-it-assessing-readability-italian/)
- [Revisiting Readability: A Unified Framework for Predicting Text Quality](https://md.research.elamur.ai/free/en/papers/pitler-2008-revisiting-readability-unified-framework-predicting/)
- [Reading Time in the Wild: An Assessment of Readability Predictors Based on Naturally-Observed Reading Times](https://md.research.elamur.ai/free/en/papers/vaals-2026-reading-time-wild-assessment-readability/)
- [On Improving the Accuracy of Readability Classification using Insights from Second Language Acquisition](https://md.research.elamur.ai/free/en/papers/vajjala-2012-improving-accuracy-readability-classification-insights/)
- [Text Readability Assessment for Second Language Learners](https://md.research.elamur.ai/free/en/papers/xia-2016-text-readability-assessment-second-language/)
- [A Monolingual Tree-based Translation Model for Sentence Simplification](https://md.research.elamur.ai/free/en/papers/zhu-2010-monolingual-tree-based-translation-model/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{schwarm2005reading,
  title = {{Reading Level Assessment Using Support Vector Machines and Statistical Language Models}},
  author = {Schwarm, Sarah and Ostendorf, Mari},
  booktitle = {Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05)},
  year = {2005},
  doi = {10.3115/1219840.1219905},
  url = {https://doi.org/10.3115/1219840.1219905}
}
```

- [← previous Pseudo-Projective Dependency Parsing](https://md.research.elamur.ai/free/en/papers/nivre-2005-pseudo-projective-dependency-parsing/)
- [next → Recognising Textual Entailment with Logical Inference](https://md.research.elamur.ai/free/en/papers/bos-2005-recognising-textual-entailment-logical-inference/)
- [↑ back to papers of 2005](https://md.research.elamur.ai/free/en/papers/2005/)

## References

27 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. R. Barzilay and N. Elhadad. Sentence alignment for monolingual comparable corpora. In Proc. of EMNLP, pages 25–32, 2003. [find](https://scholar.google.com/scholar?q=R%2E+Barzilay+and+N%2E+Elhadad%2E+Sentence+alignment+for+monolingual+comparable+corpora%2E+In+Proc%2E+of+EMNLP%2C+pages+25%E2%80%9332%2C+2003%2E) [our card](https://md.research.elamur.ai/free/en/papers/barzilay-2003-sentence-alignment-monolingual-comparable-corpora/)
2. C. Boulis and M. Ostendorf. Text classification by augmenting the bag-of-words representation with redundancy-compensated bigrams. Workshop on Feature Selection in Data Mining, in conjunction with SIAM conference on Data Mining, 2005. [find](https://scholar.google.com/scholar?q=C%2E+Boulis+and+M%2E+Ostendorf%2E+Text+classification+by+augmenting+the+bag%2Dof%2Dwords+representation+with+redundancy%2Dcompensated+bigrams%2E+Workshop+on+Feature+Selection+in+Data+Mining%2C+in+conjunction+with+SIA)
3. P. Bylsma, L. Ireland, and H. Malagon. Educating English Language Learners in Washington State. Office of the Superintendent of Public Instruction, Olympia, WA, 2003. [find](https://scholar.google.com/scholar?q=P%2E+Bylsma%2C+L%2E+Ireland%2C+and+H%2E+Malagon%2E+Educating+English+Language+Learners+in+Washington+State%2E+Office+of+the+Superintendent+of+Public+Instruction%2C+Olympia%2C+WA%2C+2003%2E)
4. J.S. Chall and E. Dale. Readability revisited: the new Dale-Chall readability formula. Brookline Books, Cambridge, Mass., 1995. [find](https://scholar.google.com/scholar?q=J%2ES%2E+Chall+and+E%2E+Dale%2E+Readability+revisited%3A+the+new+Dale%2DChall+readability+formula%2E+Brookline+Books%2C+Cambridge%2C+Mass%2E%2C+1995%2E)
5. E. Charniak. A maximum-entropy-inspired parser. In Proc. of NAACL, pages 132–139, 2000. [find](https://scholar.google.com/scholar?q=E%2E+Charniak%2E+A+maximum%2Dentropy%2Dinspired+parser%2E+In+Proc%2E+of+NAACL%2C+pages+132%E2%80%93139%2C+2000%2E) [our card](https://md.research.elamur.ai/free/en/papers/charniak-2000-maximum-entropy-inspired-parser/)
6. C. Chelba and F. Jelinek. Structured Language Modeling. Computer Speech and Language, 14(4):283-332, 2000. [find](https://scholar.google.com/scholar?q=C%2E+Chelba+and+F%2E+Jelinek%2E+Structured+Language+Modeling%2E+Computer+Speech+and+Language%2C+14%284%29%3A283%2D332%2C+2000%2E)
7. S. Chen and J. Goodman. An empirical study of smoothing techniques for language modeling. Computer Speech and Language, 13(4):359–393, 1999. [find](https://scholar.google.com/scholar?q=S%2E+Chen+and+J%2E+Goodman%2E+An+empirical+study+of+smoothing+techniques+for+language+modeling%2E+Computer+Speech+and+Language%2C+13%284%29%3A359%E2%80%93393%2C+1999%2E)
8. K. Collins-Thompson and J. Callan. A language modeling approach to predicting reading difficulty. In Proc. of HLT/NAACL, pages 193–200, 2004. [find](https://scholar.google.com/scholar?q=K%2E+Collins%2DThompson+and+J%2E+Callan%2E+A+language+modeling+approach+to+predicting+reading+difficulty%2E+In+Proc%2E+of+HLT%2FNAACL%2C+pages+193%E2%80%93200%2C+2004%2E)
9. R. Gunning. The technique of clear writing. McGraw-Hill, New York, 1952. [find](https://scholar.google.com/scholar?q=R%2E+Gunning%2E+The+technique+of+clear+writing%2E+McGraw%2DHill%2C+New+York%2C+1952%2E)
10. C.-W. Hsu et al. A practical guide to support vector classification. [http://www.csie.ntu.edu.tw/˜cjlin/papers/guide/guide.pdf](http://www.csie.ntu.edu.tw/˜cjlin/papers/guide/guide.pdf), 2003. Accessed 11/2004. [↗](http://www.csie.ntu.edu.tw/˜cjlin/papers/guide/guide.pdf)
11. T. Joachims. Text categorization with support vector machines: learning with many relevant features. In Proc. of the European Conference on Machine Learning, pages 137–142, 1998a. [find](https://scholar.google.com/scholar?q=T%2E+Joachims%2E+Text+categorization+with+support+vector+machines%3A+learning+with+many+relevant+features%2E+In+Proc%2E+of+the+European+Conference+on+Machine+Learning%2C+pages+137%E2%80%93142%2C+1998a%2E) [our card](https://md.research.elamur.ai/free/en/papers/joachims-1998-text-categorization-support-vector-machines/)
12. T. Joachims. Making large-scale support vector machine learning practical. In Advances in Kernel Methods: Support Vector Machines. B. Schölkopf, C. Burges, A. Smola, eds. MIT Press, Cambridge, MA, 1998b. [find](https://scholar.google.com/scholar?q=T%2E+Joachims%2E+Making+large%2Dscale+support+vector+machine+learning+practical%2E+In+Advances+in+Kernel+Methods%3A+Support+Vector+Machines%2E+B%2E+Scho%CC%88lkopf%2C+C%2E+Burges%2C+A%2E+Smola%2C+eds%2E+MIT+Press%2C+Cambridge%2C+MA%2C+19)
13. J.P. Kincaid, Jr., R.P. Fishburne, R.L. Rodgers, and B.S. Chisson. Derivation of new readability formulas for Navy enlisted personnel. Research Branch Report 8-75, U.S. Naval Air Station, Memphis, 1975. [find](https://scholar.google.com/scholar?q=J%2EP%2E+Kincaid%2C+Jr%2E%2C+R%2EP%2E+Fishburne%2C+R%2EL%2E+Rodgers%2C+and+B%2ES%2E+Chisson%2E+Derivation+of+new+readability+formulas+for+Navy+enlisted+personnel%2E+Research+Branch+Report+8%2D75%2C+U%2ES%2E+Naval+Air+Station%2C+Memphis%2C+197)
14. Y.-B. Lee and S.H. Myaeng. Text genre classification with genre-revealing and subject-revealing features. In Proc. of SIGIR, pages 145–150, 2002. [find](https://scholar.google.com/scholar?q=Y%2E%2DB%2E+Lee+and+S%2EH%2E+Myaeng%2E+Text+genre+classification+with+genre%2Drevealing+and+subject%2Drevealing+features%2E+In+Proc%2E+of+SIGIR%2C+pages+145%E2%80%93150%2C+2002%2E)
15. The Lexile framework for reading. [http://www.lexile.com](http://www.lexile.com), 2005. Accessed April 15, 2005. [↗](http://www.lexile.com)
16. A. Martin, G. Doddington, T. Kamm, M. Ordowski, and M. Przybocki. The DET curve in assessment of detection task performance. Proc. of Eurospeech, v. 4, pp. 1895-1898, 1997. [find](https://scholar.google.com/scholar?q=A%2E+Martin%2C+G%2E+Doddington%2C+T%2E+Kamm%2C+M%2E+Ordowski%2C+and+M%2E+Przybocki%2E+The+DET+curve+in+assessment+of+detection+task+performance%2E+Proc%2E+of+Eurospeech%2C+v%2E+4%2C+pp%2E+1895%2D1898%2C+1997%2E)
17. A. Ratnaparkhi. A maximum entropy part-of-speech tagger. In Proc. of EMNLP, pages 133–141, 1996. [find](https://scholar.google.com/scholar?q=A%2E+Ratnaparkhi%2E+A+maximum+entropy+part%2Dof%2Dspeech+tagger%2E+In+Proc%2E+of+EMNLP%2C+pages+133%E2%80%93141%2C+1996%2E)
18. B. Roark. Probabilistic top-down parsing and language modeling. Computational Linguistics, 27(2):249-276, 2001. [find](https://scholar.google.com/scholar?q=B%2E+Roark%2E+Probabilistic+top%2Ddown+parsing+and+language+modeling%2E+Computational+Linguistics%2C+27%282%29%3A249%2D276%2C+2001%2E) [our card](https://md.research.elamur.ai/free/en/papers/roark-2001-probabilistic-top-down-parsing-language/)
19. L. Si and J.P. Callan. A statistical model for scientific readability. In Proc. of CIKM, pages 574–576, 2001. [find](https://scholar.google.com/scholar?q=L%2E+Si+and+J%2EP%2E+Callan%2E+A+statistical+model+for+scientific+readability%2E+In+Proc%2E+of+CIKM%2C+pages+574%E2%80%93576%2C+2001%2E)
20. A.J. Stenner. Measuring reading comprehension with the Lexile framework. Presented at the Fourth North American Conference on Adolescent/Adult Literacy, 1996. [find](https://scholar.google.com/scholar?q=A%2EJ%2E+Stenner%2E+Measuring+reading+comprehension+with+the+Lexile+framework%2E+Presented+at+the+Fourth+North+American+Conference+on+Adolescent%2FAdult+Literacy%2C+1996%2E)
21. A. Stolcke. SRILM - an extensible language modeling toolkit. Proc. ICSLP, v. 2, pp. 901-904, 2002. [find](https://scholar.google.com/scholar?q=A%2E+Stolcke%2E+SRILM+%2D+an+extensible+language+modeling+toolkit%2E+Proc%2E+ICSLP%2C+v%2E+2%2C+pp%2E+901%2D904%2C+2002%2E)
22. U.S. Department of Education, National Center for Educational Statistics. The condition of education. [http://nces.ed.gov/programs/coe/2003/section1/indicator04.asp](http://nces.ed.gov/programs/coe/2003/section1/indicator04.asp), 2003. Accessed June 18, 2004. [↗](http://nces.ed.gov/programs/coe/2003/section1/indicator04.asp)
23. U.S. Department of Education, National Center for Educational Statistics. NCES fast facts: Bilingual education/Limited English Proficient students. [http://nces.ed.gov/fastfacts/display.asp?id=96](http://nces.ed.gov/fastfacts/display.asp?id=96), 2003. Accessed June 18, 2004. [↗](http://nces.ed.gov/fastfacts/display.asp?id=96)
24. V. Vapnik. The Nature of Statistical Learning Theory. Springer, New York, 1995. [find](https://scholar.google.com/scholar?q=V%2E+Vapnik%2E+The+Nature+of+Statistical+Learning+Theory%2E+Springer%2C+New+York%2C+1995%2E)
25. The Washington Post. [http://www.washingtonpost.com](http://www.washingtonpost.com), 2005. Accessed April 20, 2005. [↗](http://www.washingtonpost.com)
26. Weekly Reader. [http://www.weeklyreader.com](http://www.weeklyreader.com), 2004. Accessed July, 2004. [↗](http://www.weeklyreader.com)
27. Western/Pacific Literacy Network / Literacyworks. CNN SF learning resources. [http://literacynet.org/cnnsf/](http://literacynet.org/cnnsf/), 2004. Accessed June 15, 2004. [↗](http://literacynet.org/cnnsf/)