---
title: "ISSR — paper by Liu and Yen (2025) — summary and ideas"
description: "As an alternative they propose ISSR, a pipeline with three modules: a BERT-style pretrained model that proposes candidates… 33 references."
authors: ["Yu-Cheng Liu", "An-Zi Yen"]
year: 2025
venue: "arXiv:2501.03462"
source: "https://arxiv.org/abs/2501.03462"
doi: "10.48550/arxiv.2501.03462"
arxiv: "2501.03462"
licence: "CC BY-NC-ND 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-nd/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2025-issr-iterative-selection-self-review/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2025-issr-iterative-selection-self-review/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/15/) / ISSR: Iterative Selection with Self-Review for Vocabulary Test Distractor Generation

# ISSR: Iterative Selection with Self-Review for Vocabulary Test Distractor Generation

**Authors:** Yu-Cheng Liu and An-Zi Yen · **Year:** 2025 · **Venue:** arXiv:2501.03462 · **Licence:** [CC BY-NC-ND 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-nd/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2501.03462) · [PDF](https://arxiv.org/pdf/2501.03462) · [DOI 10.48550/arxiv.2501.03462](https://doi.org/10.48550/arxiv.2501.03462) · [arXiv 2501.03462](https://arxiv.org/abs/2501.03462) · [OpenAlex](https://openalex.org/W4406190830)

**Citations:** 1 citations (OpenAlex, 2026-10-14)

## Summary

The paper tackles automatic distractor generation for English gap-fill vocabulary items, using questions from Taiwan's GSAT university entrance exam. Earlier systems relied on word lists, lexical databases, hand-made rules or large training sets. They often produced options that also fit the sentence, which leaves a question with more than one right answer. The authors study GSAT items and the distributions of student responses, and they derive filtering rules from that analysis. They then test whether LLMs can produce distractors directly. Direct generation proves unstable, and it fails when many options are needed. As an alternative they propose ISSR, a pipeline with three modules:

- a BERT-style pretrained model that proposes candidates;
- an LLM selector that picks plausible ones over several small rounds;
- an LLM validator that checks each distractor by trying to solve the item itself.

ISSR needs no fine-tuning data. The self-review step removes distractors that would invalidate a question. The authors also find that the main bottleneck is the quality of the candidate pool, not the LLM's ability to choose from it.

## Key points

- The data comes from the English vocabulary section of Taiwan's GSAT, in the single-gap, four-option format. The authors also analyse how students distributed their answers across the options.
- ISSR has three parts: a candidate generator (a PLM/BERT), a distractor selector (an LLM) and a distractor validator. The validator recasts each candidate as a two-option question and lets the LLM judge whether the item stays unambiguous.
- Given a curated set of 10 candidates and asked for the top 3, LLMs selected well:
  - Llama3 8B reached F1@3 43.52% and NDCG@3 71.15%;
  - GPT-3.5 reached 35.58% and 58.60%;
  - Llama3 70B reached 33.85% and 56.65%.
- The authors tested three ways of getting 30 distractors: all at once, 10 per round, or 3 per round. Three per round scored best, with NDCG@30 of 9.82% against 6.44% for a single round of 30.
- Stated limitations:
  - running both an LLM and BERT is costly in compute;
  - the pairwise validation is slow;
  - the rules and findings are tied to GSAT;
  - output quality is capped by the quality of the candidate set.

## Interesting ideas
- Asking an LLM to choose from candidates produced by a cheaper model is more reliable than asking it to generate freely, because choosing keeps its output stable and bounded.
- Requesting a few items per round over many rounds beats one large request, since big requests appear to overload the model and lower quality.
- Self-review means testing a generated item by having the model solve it. A distractor that the solver accepts as correct is treated as a sign of ambiguity and removed.
- Validation is broken into pairwise checks: target versus one distractor at a time, posed as a binary choice. This makes each verdict simple and easy to attribute.
- The authors located the bottleneck by giving the selector a curated pool and comparing that with the full pipeline. This oracle-input test separates which stage limits quality.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): LLM teacher tiers could judge among parses or frames proposed by the witness parsers instead of writing analyses from scratch, which keeps teacher output tied to options Mova can represent.
- **I2** (learn): when using an LLM teacher to label silver data, send small batches across several rounds rather than one long list, and measure the effect against a negative control.
- **I3** (answer): when Mova builds practice school-course tests, its own reader can try each wrong option against the world of the text and discard options that turn out to be consistent with it.
- **I4** (answer): explainable answering could compare the correct answer with each alternative separately and log why each one fails, giving a per-option justification.
- **I5** (learn): feed gold trees into frames, or gold frames into world, to see which pipeline stage loses the most accuracy before spending effort on any one of them.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National Yang Ming Chiao Tung University](https://md.research.elamur.ai/free/en/universities/national-yang-ming-chiao-tung-university/) — University, Hsinchu, Taiwan · 5 works · 85 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Developmental and Educational Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/developmental-and-educational-psychology/) · 499 works

## Cite

Cite as: Yu-Cheng Liu and An-Zi Yen. 2025. ISSR: Iterative Selection with Self-Review for Vocabulary Test Distractor Generation. arXiv:2501.03462.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{liu2025issr,
  title = {{ISSR: Iterative Selection with Self-Review for Vocabulary Test Distractor Generation}},
  author = {Yu-Cheng Liu and An-Zi Yen},
  year = {2025},
  doi = {10.48550/arxiv.2501.03462},
  eprint = {2501.03462},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.03462}
}
```

- [← previous Isnad AI at IslamicEval 2025: A Rule-Based System for Identifying Religious Texts in LLM Outputs](https://md.research.elamur.ai/free/en/papers/elden-2025-isnad-ai-islamiceval-2025-rule/)
- [next → iWAN-NLP at AHaSIS 2025: A Stacked Ensemble of Arabic Transformers for Sentiment Analysis on Arabic Dialects in the Hospitality Domain](https://md.research.elamur.ai/free/en/papers/khalifa-2025-iwan-nlp-ahasis-2025-stacked/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/15/)

## References

33 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abdullah, K.I., Puteh, F., Azizan, A.R., Hamdan, N.N., Saude, S., 2013. Validation of a controlled productive Vocabulary Levels Test below the 2000-word level. System 41, 352–364. [↗](https://doi.org/10.1016/j.system.2013.03.005)
2. Blei, D.M., Ng, A.Y., Jordan, M.I., 2003. [Latent Dirichlet allocation](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/). J. Mach. Learn. Res. 3, 993–1022. [↗](https://doi.org/10.5555/944919.944937)
3. Brake, N., Schaaf, T., 2024. Comparing two model designs for clinical note generation; Is an LLM a useful evaluator of consistency?, in: Duh, K., Gomez, H., Bethard, S. (Eds.), Findings of the Association for Computational Linguistics: NAACL 2024, Association for Computational Linguistics, Mexico City, Mexico. pp. 352–363. URL: https://aclanthology.org/2024.findings-naacl.25, doi:10.18653/v1/2024. findings-naacl.25. [↗](https://doi.org/10.18653/v1/2024)
4. Breiman, L., 2001. Random forests. Machine Learning 45, 5–32. [find](https://scholar.google.com/scholar?q=Breiman%2C+L%2E%2C+2001%2E+Random+forests%2E+Machine+Learning+45%2C+5%E2%80%9332%2E)
5. Burges, C.J., 2010. From RankNet to LambdaRank to LambdaMART: An overview. Learning 11, 81. [find](https://scholar.google.com/scholar?q=Burges%2C+C%2EJ%2E%2C+2010%2E+From+RankNet+to+LambdaRank+to+LambdaMART%3A+An+overview%2E+Learning+11%2C+81%2E)
6. Chiang, S.H., Wang, S.C., Fan, Y.C., 2022. CDGP: Automatic cloze distractor generation based on pre-trained language model, in: Goldberg, Y., Kozareva, Z., Zhang, Y. (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, Association for Computational Linguistics, Abu Dhabi, United Arab Emirates. pp. 5835–5840. URL: https://aclanthology.org/2022.findings-emnlp.429, doi:10.18653/ v1/2022.findings-emnlp.429. [↗](https://aclanthology.org/2022.findings-emnlp.429/)
7. COBUILD, C., 1995. Collins COBUILD English dictionary. HarperCollins. [find](https://scholar.google.com/scholar?q=COBUILD%2C+C%2E%2C+1995%2E+Collins+COBUILD+English+dictionary%2E+HarperCollins%2E)
8. COBUILD, C., 2001. Collins COBUILD English Dictionary for Advanced Learners. 3rd ed., HarperCollins, Glasgow. [find](https://scholar.google.com/scholar?q=COBUILD%2C+C%2E%2C+2001%2E+Collins+COBUILD+English+Dictionary+for+Advanced+Learners%2E+3rd+ed%2E%2C+HarperCollins%2C+Glasgow%2E)
9. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al., 2024. [The Llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783 . [↗](https://arxiv.org/abs/2407.21783)
10. Fellbaum, C., 1998. WordNet: An Electronic Lexical Database. MIT Press 2, 678–686. [find](https://scholar.google.com/scholar?q=Fellbaum%2C+C%2E%2C+1998%2E+WordNet%3A+An+Electronic+Lexical+Database%2E+MIT+Press+2%2C+678%E2%80%93686%2E)
11. Freund, Y., Schapire, R.E., 1997. A decision-theoretic generalization of online learning and an application to boosting. Journal of Computer and System Sciences 55, 119–139. [find](https://scholar.google.com/scholar?q=Freund%2C+Y%2E%2C+Schapire%2C+R%2EE%2E%2C+1997%2E+A+decision%2Dtheoretic+generalization+of+online+learning+and+an+application+to+boosting%2E+Journal+of+Computer+and+System+Sciences+55%2C+119%E2%80%93139%2E)
12. Ha, H.T., 2022. Issues in matching learners to their reading levels using receptive vocabulary knowledge – A closer look at the vocabulary levels tests and scoring methods. Heliyon 8. [↗](https://doi.org/10.1016/j.heliyon.2022.e10244)
13. Heaton, J.B., 1988. Writing English Language Tests. Longman. [find](https://scholar.google.com/scholar?q=Heaton%2C+J%2EB%2E%2C+1988%2E+Writing+English+Language+Tests%2E+Longman%2E)
14. Ishikawa, S., Uemura, T., Kaneda, M., Shimizu, S., Sugimori, N., Tono, Y., Murata, M., 2003. JACET8000: JACET list of 8000 basic words. Tokyo: JACET 3. [find](https://scholar.google.com/scholar?q=Ishikawa%2C+S%2E%2C+Uemura%2C+T%2E%2C+Kaneda%2C+M%2E%2C+Shimizu%2C+S%2E%2C+Sugimori%2C+N%2E%2C+Tono%2C+Y%2E%2C+Murata%2C+M%2E%2C+2003%2E+JACET8000%3A+JACET+list+of+8000+basic+words%2E+Tokyo%3A+JACET+3%2E)
15. Islam, R., Moushi, O.M., 2024. GPT-4o: The cutting-edge advancement in multimodal LLM. Authorea Preprints . [find](https://scholar.google.com/scholar?q=Islam%2C+R%2E%2C+Moushi%2C+O%2EM%2E%2C+2024%2E+GPT%2D4o%3A+The+cutting%2Dedge+advancement+in+multimodal+LLM%2E+Authorea+Preprints+%2E)
16. Jalali, M., Luo, Y., Caulfield, L., Sauter, E., Nefedov, A., Wöll, C., 2024. Large language models in electronic laboratory notebooks: Transforming materials science research workflows. Materials Today Communications 40, 109801. URL: https://www.sciencedirect.com/science/article/pii/S2352492824017823, doi:https://doi.org/10.1016/j.mtcomm.2024.109801. [↗](https://doi.org/10.1016/j.mtcomm.2024.109801)
17. Jiang, S., Lee, J., 2017. Distractor generation for Chinese fill-in-the-blank items, in: Tetreault, J., Burstein, J., Leacock, C., Yannakoudakis, H. (Eds.), Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, Association for Computational Linguistics, Copenhagen, Denmark. pp. 143–148. URL: https://aclanthology.org/W17-5015, doi:10.18653/v1/W17-5015. [↗](https://doi.org/10.18653/v1/w17-5015)
18. Kim, D., Wang, H., Oh, A., 2013. Context-dependent conceptualization, in: Twenty-Third International Joint Conference on Artificial Intelligence, Citeseer. [find](https://scholar.google.com/scholar?q=Kim%2C+D%2E%2C+Wang%2C+H%2E%2C+Oh%2C+A%2E%2C+2013%2E+Context%2Ddependent+conceptualization%2C+in%3A+Twenty%2DThird+International+Joint+Conference+on+Artificial+Intelligence%2C+Citeseer%2E)
19. Liang, C., Yang, X., Dave, N., Wham, D., Pursel, B., Giles, C.L., 2018. Distractor generation for multiple choice questions using learning to rank, in: Tetreault, J., Burstein, J., Kochmar, E., Leacock, C., Yannakoudakis, H. (Eds.), Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, Association for Computational Linguistics, New Orleans, Louisiana. pp. 284–290. URL: https://aclanthology.org/W18-0533, doi:10.18653/v1/W18-0533. [↗](https://doi.org/10.18653/v1/w18-0533)
20. Liu, Y., Liu, J., Shi, X., Cheng, Q., Lu, W., 2024. Let’s learn step by step: Enhancing in-context learning ability with curriculum learning. arXiv preprint arXiv:2402.10738 . [↗](https://arxiv.org/abs/2402.10738)
21. Morea, N., Kasprowicz, R.E., Morrison, A., Silvestri, C., 2024. Diverse population, homogenous ability: The development of a new receptive vocabulary size test for young language learners in England using Rasch analysis. Research Methods in Applied Linguistics 3, 100166. [↗](https://doi.org/10.1016/j.rmal.2024.100166)
22. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., Lowe, R., 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems 35, 27730–27744.
23. Pennington, J., Socher, R., Manning, C.D., 2014. [GloVe: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/), in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543.
24. Ren, S., Zhu, K.Q., 2021. Knowledge-driven distractor generation for cloze-style multiple choice questions, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4339–4347. [find](https://scholar.google.com/scholar?q=Ren%2C+S%2E%2C+Zhu%2C+K%2EQ%2E%2C+2021%2E+Knowledge%2Ddriven+distractor+generation+for+cloze%2Dstyle+multiple+choice+questions%2C+in%3A+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+pp%2E+4339%E2%80%934347%2E)
25. Rubin, O., Herzig, J., Berant, J., 2022. [Learning to retrieve prompts for in-context learning](https://md.research.elamur.ai/free/en/papers/rubin-2021-learning-retrieve-prompts-context-learning/), in: Carpuat, M., de Marneffe, M.C., Meza Ruiz, I.V. (Eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Seattle, United States. pp. 2655–2671. URL: https://aclanthology.org/2022. naacl-main.191, doi:10.18653/v1/2022.naacl-main.191. [↗](https://doi.org/10.18653/v1/2022.naacl-main.191)
26. Susanti, Y., Tokunaga, T., Nishikawa, H., Obari, H., 2018. Automatic distractor generation for multiple-choice English vocabulary questions. Research and Practice in Technology Enhanced Learning 13, 15. [↗](https://doi.org/10.1186/s41039-018-0082-z)
27. Tseng, W.T., 2016. Measuring English vocabulary size via computerized adaptive testing. Computers & Education 97, 69–85. [↗](https://doi.org/10.1016/j.compedu.2016.02.018)
28. Viola, P., Jones, M., 2001. Rapid object detection using a boosted cascade of simple features, in: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, pp. I–I. doi:10.1109/CVPR.2001.990517. [↗](https://doi.org/10.1109/cvpr.2001.990517)
29. Wang, J., Yu, L., Zhang, W., Gong, Y., Xu, Y., Wang, B., Zhang, P., Zhang, D., 2017. IRGAN: A minimax game for unifying generative and discriminative information retrieval models, in: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Association for Computing Machinery, New York, NY, USA. p. 515–524. URL: https://doi.org/10.1145/3077136.3080786, doi:10.1145/3077136.3080786. [↗](https://doi.org/10.1145/3077136.3080786)
30. Wei, J., Hou, L., Lampinen, A., Chen, X., Huang, D., Tay, Y., Chen, X., Lu, Y., Zhou, D., Ma, T., Le, Q., 2023. Symbol tuning improves in-context learning in language models, in: Bouamor, H., Pino, J., Bali, K. (Eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Singapore. pp. 968–979. URL: https://aclanthology.org/2023.emnlp-main.61, doi:10.18653/v1/2023.emnlp-main.61. [↗](https://doi.org/10.18653/v1/2023.emnlp-main.61)
31. Wu, W., Li, H., Wang, H., Zhu, K.Q., 2012. Probase: A probabilistic taxonomy for text understanding, in: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, pp. 481–492. [find](https://scholar.google.com/scholar?q=Wu%2C+W%2E%2C+Li%2C+H%2E%2C+Wang%2C+H%2E%2C+Zhu%2C+K%2EQ%2E%2C+2012%2E+Probase%3A+A+probabilistic+taxonomy+for+text+understanding%2C+in%3A+Proceedings+of+the+2012+ACM+SIGMOD+International+Conference+on+Management+of+Data%2C+pp%2E+481%E2%80%93492%2E)
32. Xie, Q., Lai, G., Dai, Z., Hovy, E., 2018. Large-scale cloze test dataset created by teachers, in: Riloff, E., Chiang, D., Hockenmaier, J., Tsujii, J. (Eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Brussels, Belgium. pp. 2344–2356. URL: https://aclanthology.org/D18-1257, doi:10.18653/v1/D18-1257. [↗](https://doi.org/10.18653/v1/d18-1257)
33. Yorio, C.A., 1971. Some sources of reading problems for foreign-language learners. Language Learning 21, 107–115. [find](https://scholar.google.com/scholar?q=Yorio%2C+C%2EA%2E%2C+1971%2E+Some+sources+of+reading+problems+for+foreign%2Dlanguage+learners%2E+Language+Learning+21%2C+107%E2%80%93115%2E)

*Updated 1 November 2026*