---
title: "Auto-Select Reading Passages in English Assessment Tests?"
description: "The paper asks whether reading passages for standardized English exams can be chosen automatically. 14 references, cited by 1 paper here."
authors: ["Bruce W. Lee", "Jason H. Lee"]
year: 2022
venue: "arXiv:2205.06961"
source: "https://arxiv.org/abs/2205.06961"
doi: "10.48550/arxiv.2205.06961"
arxiv: "2205.06961"
licence: "CC BY-NC-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lee-2022-auto-select-reading-passages-english/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lee-2022-auto-select-reading-passages-english/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/6/) / Auto-Select Reading Passages in English Assessment Tests?

# Auto-Select Reading Passages in English Assessment Tests?

**Authors:** [Bruce W. Lee](https://md.research.elamur.ai/free/en/authors/bruce-w-lee/) and Jason H. Lee · **Year:** 2022 · **Venue:** arXiv:2205.06961 · **Licence:** [CC BY-NC-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2205.06961) · [PDF](https://arxiv.org/pdf/2205.06961) · [DOI 10.48550/arxiv.2205.06961](https://doi.org/10.48550/arxiv.2205.06961) · [arXiv 2205.06961](https://arxiv.org/abs/2205.06961) · [OpenAlex](https://openalex.org/W4280532491)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks whether reading passages for standardized English exams can be chosen automatically. Its running example is South Korea's CSAT. The full expert selection process is hard to imitate, so the authors take a shortcut. They start from a "gold instance," a passage that already appeared in a real test, and search for passages that resemble it on several dimensions. They built a database of about a thousand paragraphs by running OCR on the source articles of 2021–2022 CSAT passages. Each paragraph was described with spaCy counts, a grade-level word difficulty lexicon (CoKEC-word), and SBERT embeddings. A chain of threshold filters then keeps only candidates close to the gold passage. Three Korean EFL practitioners rated filtered candidates mixed with random ones. On average they found the filtered passages more suitable. However, no single feature correlated with the suitability ratings, including length, sentence count, and CEFR level. The authors conclude that suitability arises from combinations of features. This makes simple regression unpromising, while similarity to gold remains a workable proxy.

## Key points
- **Database:** 1,044 passages, 10,651 sentences, roughly 2.2e5 words. The passages were OCR'd with Google Vision API, split into paragraphs by hand, and only lightly cleaned.
- **Extractors:**
  - spaCy `en_core_web_sm` (v3.2.1) for sentence and word counts.
  - CoKEC-word, which places 30,608 words into 6 grade bands (A–F).
  - SBERT `all-MiniLM-L6-v2` for semantic embeddings.
- **Filters:** five applied in sequence:
  - sentence count;
  - words per sentence;
  - highest word difficulty;
  - share of grade-9+ words;
  - cosine similarity to the gold passage.
  
  Each filter has a hand-tuned α, chosen to yield about 5 candidates per gold passage.
- **Evaluation design:**
  - 10 gold passages, each with 5 filtered and 5 random "dummy" candidates, shuffled together.
  - Raters were 2 teachers with 10+ years of experience and 1 recent CSAT taker.
  - Each candidate received a 5-point suitability rating.
- **Results:**
  - Mean summed scores were 15.5 for filtered and 12.8 for dummy passages (+2.7). Filtered candidates won in 8 of 10 trials.
  - Correlations of the scores with #word, #sent, and CEFR level (from the RoBERTa-RF-T1 classifier) were negligible.

## Interesting ideas
- When a selection process is hard to model directly, anchor on exemplars that already passed it and retrieve look-alikes, so you avoid predicting the target property itself.
- Retrieval is done with a cascade of plain, interpretable threshold filters (length, difficulty, semantics), so every rejection can be traced to a specific failed condition.
- Random dummy items from the same pool are shuffled in with system outputs before blind rating, which gives a built-in negative control.
- Proposed objective check: hide the gold paragraph among its neighbouring paragraphs from the same article and test whether a model can pick it out.
- A passage should stand on its own, so the authors plan to detect dependence on earlier text, tables, or figures.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (answer): New practice texts for school-course tests could be gathered by retrieving passages similar to known good exam texts, instead of training a "suitability" scorer.
- **I2** (learn): Candidate texts for silver-data harvesting could be selected with readable, staged filters, keeping each inclusion or exclusion explainable.
- **I3** (learn): Evaluations of generated frames or answers could hide random distractors among real outputs so judges' baseline leniency becomes measurable.
- **I4** (answer): A similar "find the original among siblings" task could test whether readers recognise which paragraph is self-contained and question-worthy.
- **I5** (frames): Coreference output could flag passages with unresolved anaphora or references to figures as not self-contained before they are used as QA inputs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works

## Cite

Cite as: Bruce W. Lee and Jason H. Lee. 2022. Auto-Select Reading Passages in English Assessment Tests?. arXiv:2205.06961.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{lee2022auto,
  title = {{Auto-Select Reading Passages in English Assessment Tests?}},
  author = {Bruce W. Lee and Jason H. Lee},
  year = {2022},
  doi = {10.48550/arxiv.2205.06961},
  eprint = {2205.06961},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2205.06961}
}
```

- [← previous An End-to-End OCR Framework for Robust Arabic-Handwriting Recognition using a Novel Transformers-based Model and an Innovative 270 Million-Words Multi-Font Corpus of Classical Arabic with Diacritics](https://md.research.elamur.ai/free/en/papers/mostafa-2022-end-end-ocr-framework-robust/)
- [next → Coreference Annotation of an Arabic Corpus using a Virtual World Game](https://md.research.elamur.ai/free/en/papers/aliady-2022-coreference-annotation-arabic-corpus-virtual/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/6/)

## References

14 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. College-Board. 2017. Sat® suite of assessments technical manual: Characteristics of the sat. [find](https://scholar.google.com/scholar?q=College%2DBoard%2E+2017%2E+Sat%C2%AE+suite+of+assessments+technical+manual%3A+Characteristics+of+the+sat%2E)
2. Kevyn Collins-Thompson. 2014. Computational assessment of text readability: A survey of current and future research. ITL-International Journal of Applied Linguistics, 165(2):97–135. [find](https://scholar.google.com/scholar?q=Kevyn+Collins%2DThompson%2E+2014%2E+Computational+assessment+of+text+readability%3A+A+survey+of+current+and+future+research%2E+ITL%2DInternational+Journal+of+Applied+Linguistics%2C+165%282%29%3A97%E2%80%93135%2E)
3. Educational-Testing-Service. 2010. framework and test development. Toefl ibt® test Wael H Gomaa, Aly A Fahmy, et al. 2013. A survey of text similarity approaches. international journal of Computer Applications, 68(13):13–18. [find](https://scholar.google.com/scholar?q=Educational%2DTesting%2DService%2E+2010%2E+framework+and+test+development%2E+Toefl+ibt%C2%AE+test+Wael+H+Gomaa%2C+Aly+A+Fahmy%2C+et+al%2E+2013%2E+A+survey+of+text+similarity+approaches%2E+international+journal+of+Computer+App)
4. Matthew Honnibal and Ines Montani. 2017. spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. To appear. [find](https://scholar.google.com/scholar?q=Matthew+Honnibal+and+Ines+Montani%2E+2017%2E+spacy+2%3A+Natural+language+understanding+with+bloom+embeddings%2C+convolutional+neural+networks+and+incremental+parsing%2E+To+appear%2E)
5. Ching-Kun Hsu, Gwo-Jen Hwang, and Chih-Kai Chang. 2010. Development of a reading material recommendation system based on a knowledge engineering approach. Computers & Education, 55(1):76–83. [↗](https://doi.org/10.1016/j.compedu.2009.12.004)
6. KICE. 2014. What is csat (college scholastic ability test)? [find](https://scholar.google.com/scholar?q=KICE%2E+2014%2E+What+is+csat+%28college+scholastic+ability+test%29%3F)
7. Chin-Hwa Kuo, Chen-Chung Chi, et al. 2014. Designing a reading material recommendation system for efl learners. Journal of Applied Science and Engineering, 17(4):371–382. [↗](https://doi.org/10.6180/jase.2014.17.4.04)
8. Bruce W. Lee, Yoo Sung Jang, and Jason Lee. 2021. [Pushing on text readability assessment: A transformer meets handcrafted linguistic features](https://md.research.elamur.ai/free/en/papers/lee-2021-pushing-text-readability-assessment-transformer/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10669– 10686, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
9. Bruce W. Lee and Jason Lee. 2020. LXPER index 2.0: Improving text readability assessment model for L2 English students in Korea. In Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications, pages 20– 24, Suzhou, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bruce+W%2E+Lee+and+Jason+Lee%2E+2020%2E+LXPER+index+2%2E0%3A+Improving+text+readability+assessment+model+for+L2+English+students+in+Korea%2E+In+Proceedings+of+the+6th+Workshop+on+Natural+Language+Processing+Techn)
10. Dimas Wibisono Prakoso, Asad Abdi, and Chintan Amrit. 2021. Short text similarity measurement methods: a review. Soft Computing, 25(6):4699–4723. [↗](https://doi.org/10.1007/s00500-020-05479-2)
11. Nils Reimers and Iryna Gurevych. 2019. [Sentence-BERT: Sentence embeddings using Siamese BERT-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d19-1410)
12. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776–5788. [find](https://scholar.google.com/scholar?q=Wenhui+Wang%2C+Furu+Wei%2C+Li+Dong%2C+Hangbo+Bao%2C+Nan+Yang%2C+and+Ming+Zhou%2E+2020%2E+Minilm%3A+Deep+self%2Dattention+distillation+for+task%2Dagnostic+compression+of+pre%2Dtrained+transformers%2E+Advances+in+Neural+Inform)
13. Paul A Westrick, Jessica P Marini, Linda Young, Helen Ng, Doron Shmueli, and Emily J Shaw. 2019. Validity of the sat® for predicting first-year grades and retention to the second year. College Board Research Paper. [find](https://scholar.google.com/scholar?q=Paul+A+Westrick%2C+Jessica+P+Marini%2C+Linda+Young%2C+Helen+Ng%2C+Doron+Shmueli%2C+and+Emily+J+Shaw%2E+2019%2E+Validity+of+the+sat%C2%AE+for+predicting+first%2Dyear+grades+and+retention+to+the+second+year%2E+College+Board+R)
14. Menglin Xia, Ekaterina Kochmar, and Ted Briscoe. 2016. [Text readability assessment for second language learners](https://md.research.elamur.ai/free/en/papers/xia-2016-text-readability-assessment-second-language/). In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, pages 12–22, San Diego, CA. Association for Computational Linguistics.

*Updated 24 October 2026*